Decoding method and apparatus with candidate motion vectors

ABSTRACT

An image coding method includes: determining a maximum number of merging candidates; deriving first merging candidates based on blocks spatially or temporally neighboring a current block; determining whether or not a total number of the first merging candidates is smaller than the maximum number; deriving, by making a combination out of the derived first merging candidates, a second merging candidate for bi-directional prediction when it is determined that the total number of the first merging candidates is smaller than the maximum number; selecting a merging candidate to be used for the coding of the current block from the first merging candidates and the second merging candidate; and coding, using the determined maximum number, an index for identifying the selected merging candidate, and attaching the coded index to the bitstream.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the benefit of U.S. Provisional PatentApplication No. 61/491,549 filed May 31, 2011. The entire disclosure ofthe above-identified application, including the specification, drawingsand claims are incorporated herein by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to an image coding method and an imagedecoding method.

BACKGROUND ART

Generally, in coding processing of a moving picture, the amount ofinformation is reduced by compression for which redundancy of a movingpicture in spatial direction and temporal direction is made use of.Generally, conversion to a frequency domain is performed as a method inwhich redundancy in spatial direction is made use of, and coding usingprediction between pictures (the prediction is hereinafter referred toas inter prediction) is performed as a method of compression for whichredundancy in temporal direction is made use of. In the inter predictioncoding, a current picture is coded using, as a reference picture, acoded picture which precedes or follows the current picture in order ofdisplay time. Subsequently, a motion vector is derived by performingmotion estimation on the current picture with reference to the referencepicture. Then, redundancy in temporal direction is removed using acalculated difference between picture data of the current picture andprediction picture data which is obtained by motion compensation basedon the derived motion vector (see Non-patent Literature 1, for example).Here, in the motion estimation, difference values between current blocksin the current picture and blocks in the reference picture arecalculated, and a block having the smallest difference value in thereference picture is determined as a reference block. Then, a motionvector is estimated from the current block and the reference block.

CITATION LIST Non Patent Literature

-   [Non-patent Literature 1] ITU-T Recommendation H.264 “Advanced video    coding for generic audiovisual services”, March 2010-   [Non-patent Literature 2] JCT-VC, “WD3: Working Draft 3 of    High-Efficiency Video Coding”, JCTVC-E603, March 2011

SUMMARY OF INVENTION Technical Problem

It is still desirable to increase coding efficiency for image coding anddecoding in which inter prediction is used, beyond the above-describedconventional technique.

In view of this, the object of the present invention is to provide animage coding method and an image decoding method with which codingefficiency in image coding and image decoding using inter prediction isincreased.

Solution to Problem

An image coding method according to an aspect of the present inventionis a method for coding an image on a block-by-block basis to generate abitstream, and includes: determining a maximum number of mergingcandidates each of which is a combination of a prediction direction, amotion vector, and a reference picture index for use in coding of acurrent block;

deriving a plurality of first merging candidates based on predictiondirections, motion vectors, and reference picture indexes used in codingof blocks spatially or temporally neighboring the current block;determining whether or not a total number of the derived first mergingcandidates is smaller than the maximum number; deriving, by making acombination out of the derived first merging candidates, a secondmerging candidate for bi-directional prediction when it is determinedthat the total number of the derived first merging candidates is smallerthan the maximum number; selecting a merging candidate to be used forthe coding of the current block from the derived first mergingcandidates and the derived second merging candidate; and coding, usingthe determined maximum number, an index for identifying the selectedmerging candidate, and attaching the coded index to the bitstream.

It should be noted that these general or specific aspects can beimplemented as a system, a method, an integrated circuit, a computerprogram, a computer-readable recording medium such as a compact discread-only memory (CD-ROM), or as any combination of a system, a method,an integrated circuit, a computer program, and a computer-readablerecording medium.

Advantageous Effects of Invention

According to an aspect of the present invention, coding efficiency inimage coding and decoding using inter prediction can be increased.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the invention willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present invention. In the Drawings:

FIG. 1A is a diagram for illustrating an exemplary reference picturelist for a B picture;

FIG. 1B is a diagram for illustrating an exemplary reference picturelist of a prediction direction 0 for a B picture;

FIG. 1C is a diagram for illustrating an exemplary reference picturelist of a prediction direction 1 for a B picture;

FIG. 2 is a diagram for illustrating motion vectors for use in thetemporal motion vector prediction mode;

FIG. 3 shows an exemplary motion vector of a neighboring block for usein the merging mode;

FIG. 4 is a diagram for illustrating an exemplary merging blockcandidate list;

FIG. 5 shows a relationship between the size of a merging blockcandidate list and bit sequences assigned to merging block candidateindexes;

FIG. 6 is a flowchart showing an example of a process for coding whenthe merging mode is used;

FIG. 7 is a flowchart showing a process for decoding using the mergingmode;

FIG. 8 shows syntax for attachment of merging block candidate indexes toa bitstream;

FIG. 9 is a block diagram showing a configuration of an image codingapparatus according to Embodiment 1;

FIG. 10 is a flowchart showing processing operations of the image codingapparatus according to Embodiment 1;

FIG. 11 shows an exemplary merging block candidate list according toEmbodiment 1;

FIG. 12 is a flowchart illustrating a process for calculating mergingblock candidates and the size of a merging block candidate listaccording to Embodiment 1;

FIG. 13 is a flowchart illustrating a process for determining whether ornot a merging block candidate is a usable-for-merging candidate andupdating the total number of usable-for-merging candidates according toEmbodiment 1;

FIG. 14 is a flowchart illustrating a process for adding a combinedmerging block candidate according to Embodiment 1;

FIG. 15 is a flowchart illustrating a process for determining whether ornot there is a combined merging block candidate according to Embodiment1;

FIG. 16 is a flowchart illustrating a process for selecting a mergingblock candidate according to Embodiment 1;

FIG. 17 is a block diagram showing a configuration of an image codingapparatus according to Embodiment 2;

FIG. 18 is a flowchart showing processing operations of the image codingapparatus according to Embodiment 2;

FIG. 19 is a block diagram showing a configuration of an image decodingapparatus according to Embodiment 3;

FIG. 20 is a flowchart showing processing operations of the imagedecoding apparatus according to Embodiment 3;

FIG. 21 is a block diagram showing a configuration of an image decodingapparatus according to Embodiment 4;

FIG. 22 is a flowchart showing processing operations of the imagedecoding apparatus according to Embodiment 4;

FIG. 23 is a block diagram showing a configuration of an image codingapparatus according to Embodiment 5;

FIG. 24 is a flowchart showing processing operations of the image codingapparatus according to Embodiment 5;

FIG. 25 shows an exemplary merging block candidate list according toEmbodiment 5;

FIG. 26 is a flowchart illustrating a process for calculating mergingblock candidates and the size of a merging block candidate listaccording to Embodiment 5;

FIG. 27 is a flowchart illustrating a process for updating a totalnumber of usable-for-merging candidates according to Embodiment 5,

FIG. 28 is a flowchart illustrating a process for adding a new candidateaccording to Embodiment 5;

FIG. 29 is a block diagram showing a configuration of an image codingapparatus according to Embodiment 6;

FIG. 30 is a flowchart showing processing operations of the image codingapparatus according to Embodiment 6;

FIG. 31 is a block diagram showing a configuration of an image decodingapparatus according to Embodiment 7;

FIG. 32 is a flowchart showing processing operations of the imagedecoding apparatus according to Embodiment 7;

FIG. 33 is a flowchart illustrating a process for setting the size of amerging block candidate list according to Embodiment 7;

FIG. 34 is a flowchart illustrating a process for calculating a mergingblock candidate according to Embodiment 7;

FIG. 35 shows syntax for attachment of merging block candidate indexesto a bitstream;

FIG. 36 shows exemplary syntax in the case where the size of a mergingblock candidate list is fixed at the maximum value of the total numberof merging block candidates;

FIG. 37 is a block diagram showing a configuration of an image decodingapparatus according to Embodiment 8;

FIG. 38 is a flowchart showing processing operations of the imagedecoding apparatus according to Embodiment 8;

FIG. 39 shows an overall configuration of a content providing system forimplementing content distribution services;

FIG. 40 shows an overall configuration of a digital broadcasting system;

FIG. 41 shows a block diagram illustrating an example of a configurationof a television;

FIG. 42 is a block diagram illustrating an example of a configuration ofan information reproducing/recording unit that reads and writesinformation from and on a recording medium that is an optical disk;

FIG. 43 shows an example of a configuration of a recording medium thatis an optical disk;

FIG. 44A shows an example of a cellular phone;

FIG. 44B is a block diagram showing an example of a configuration of acellular phone;

FIG. 45 illustrates a structure of multiplexed data;

FIG. 46 schematically shows how each stream is multiplexed inmultiplexed data;

FIG. 47 shows how a video stream is stored in a stream of PES packets inmore detail;

FIG. 48 shows a structure of TS packets and source packets in themultiplexed data;

FIG. 49 shows a data structure of a PMT;

FIG. 50 shows an internal structure of multiplexed data information;

FIG. 51 shows an internal structure of stream attribute information;

FIG. 52 shows steps for identifying video data;

FIG. 53 is a block diagram showing an example of a configuration of anintegrated circuit for implementing the moving picture coding method andthe moving picture decoding method according to each of embodiments;

FIG. 54 shows a configuration for switching between driving frequencies;

FIG. 55 shows steps for identifying video data and switching betweendriving frequencies;

FIG. 56 shows an example of a look-up table in which video datastandards are associated with driving frequencies;

FIG. 57A is a diagram showing an example of a configuration for sharinga module of a signal processing unit; and

FIG. 57B is a diagram showing another example of a configuration forsharing a module of the signal processing unit.

DESCRIPTION OF EMBODIMENTS Underlying Knowledge Forming Basis of thePresent Invention

In a moving picture coding scheme already standardized, which isreferred to as H.264, three picture types of I picture, P picture, and Bpicture are used for reduction of the amount of information bycompression.

The I picture is not coded by inter prediction coding. Specifically, theI picture is coded by prediction within the picture (the prediction ishereinafter referred to as intra prediction). The P picture is coded byinter prediction coding with reference to one coded picture preceding orfollowing the current picture in order of display time. The B picture iscoded by inter prediction coding with reference to two coded picturespreceding and following the current picture in order of display time.

In inter prediction coding, a reference picture list for identifying areference picture is generated. In a reference picture list, referencepicture indexes are assigned to coded reference pictures to bereferenced in inter prediction. For example, two reference picture lists(L0, L1) are generated for a B picture because it can be coded withreference to two pictures.

FIG. 1A is a diagram for illustrating an exemplary reference picturelist for a B picture. FIG. 1B shows an exemplary reference picture list0 (L0) for a prediction direction 0 in bi-directional prediction. In thereference picture list 0, the reference picture index 0 having a valueof 0 is assigned to a reference picture 0 with a display order number 2.The reference picture index 0 having a value of 1 is assigned to areference picture 1 with a display order number 1. The reference pictureindex 0 having a value of 2 is assigned to a reference picture 2 with adisplay order number 0. In other words, the shorter the temporaldistance of a reference picture from the current picture, the smallerthe reference picture index assigned to the reference picture.

On the other hand, FIG. 1C shows an exemplary reference picture list 1(L1) for a prediction direction 1 in bi-directional prediction. In thereference picture list 1, the reference picture index 1 having a valueof 0 is assigned to a reference picture 1 with a display order number 1.The reference picture index 1 having a value of 1 is assigned to areference picture 0 with a display order number 2. The reference pictureindex 2 having a value of 2 is assigned to a reference picture 2 with adisplay order number 0.

In this manner, it is possible to assign reference picture indexeshaving values different between prediction directions to a referencepicture (the reference pictures 0 and 1 in FIG. 1A) or to assign thereference picture index having the same value for both directions to areference picture (the reference picture 2 in FIG. 1A).

In a moving picture coding method referred to as H.264 (see Non-patentLiterature 1), a motion vector estimation mode is available as a codingmode for inter prediction of each current block in a B picture. In themotion vector estimation mode, a difference value between picture dataof a current block and prediction picture data and a motion vector usedfor generating the prediction picture data are coded. In addition, inthe motion vector estimation mode, bi-directional prediction anduni-directional prediction can be selectively performed. Inbi-directional prediction, a prediction picture is generated withreference to two coded pictures one of which precedes a current pictureto be coded and the other of which follows the current picture. Inuni-directional prediction, a prediction picture is generated withreference to one coded picture preceding or following a current pictureto be coded.

Furthermore, in the moving picture coding method referred to as H.264, acoding mode referred to as a temporal motion vector prediction mode canbe selected for derivation of a motion vector in coding of a B picture.The inter prediction coding method performed in the temporal motionvector prediction mode will be described below using FIG. 2. FIG. 2 is adiagram for illustrating motion vectors for use in the temporal motionvector prediction mode. Specifically, FIG. 2 shows a case where a blocka in a picture B2 is coded in temporal motion vector prediction mode.

In the coding, a motion vector vb is used which has been used in codinga block b located in the same position in a picture P3, which is areference picture following the picture B2, as the position of the blocka in the picture B2 (in the case, the block b is hereinafter referred toas a co-located block of the block a). The motion vector vb is a motionvector used in coding the block b with reference to the picture P1.

Two reference blocks for the block a are obtained from a forwardreference picture and a backward reference picture, that is, a pictureP1 and a picture P3 using motion vectors parallel to the motion vectorvb. Then, the block a is coded by bi-directional prediction based on thetwo obtained reference blocks. Specifically, in the coding of the blocka, a motion vector va1 is used to reference the picture P1, and a motionvector vat is used to reference the picture P3.

In addition, a merging mode is discussed as an inter prediction mode forcoding of each current block in a B picture or a P picture (seeNon-patent Literature 2). In the merging mode, a current block is codedusing a prediction direction, a motion vector, and a reference pictureindex which are duplications of those used in coding a neighboring blockof the current block. At this time, the index and others of theneighboring block used for the copying are attached to a bitstream sothat the motion direction, motion vector, and reference picture indexused in the coding can be selected in decoding. A concrete example forit is given below with reference to FIG. 3.

FIG. 3 shows an exemplary motion vector of a neighboring block for usein the merging mode. In FIG. 3, a neighboring block A is a coded blocklocated on the immediate left of a current block. A neighboring block Bis a coded block located immediately above the current block. Aneighboring block C is a coded block located immediately right above thecurrent block. A neighboring block D is a coded block locatedimmediately left below the current block.

The neighboring block A is a block coded by uni-directional predictionin the prediction direction 0. The neighboring block A has a motionvector MvL0_A having the prediction direction 0 as a motion vector withrespect to a reference picture indicated by a reference picture indexRefL0_A of the prediction direction 0. Here, MvL0 indicates a motionvector which references a reference picture specified in a referencepicture list 0 (L0). MvL1 indicates a motion vector which references areference picture specified in a reference picture list 1 (L1).

The neighboring block B is a block coded by uni-directional predictionin the prediction direction 1. The neighboring block B has a motionvector MvL1_B having the prediction direction 1 as a motion vector withrespect to a reference picture indicated by a reference picture indexRefL1_B of the prediction direction 1.

The neighboring block C is a block coded by intra prediction.

The neighboring block D is a block coded by uni-directional predictionin the prediction direction 0. The neighboring block D has a motionvector MvL0_D having the prediction direction 0 as a motion vector withrespect to a reference picture indicated by a reference picture indexRefL0_D of the prediction direction 0.

In this case, for example, a combination of a prediction direction, amotion vector, and a reference picture index with which the currentblock can be coded with the highest coding efficiency is selected as aprediction direction, a motion vector, and a reference picture index ofthe current block from the prediction directions, motion vectors andreference picture indexes of the neighboring blocks A to D, and aprediction direction, a motion vector, and a reference picture indexwhich are calculated using a co-located block in temporal motion vectorprediction mode. Then, a merging block candidate index indicating theblock having the selected combination of a prediction direction, amotion vector, and a reference picture index is attached to a bitstream.

For example, when the neighboring block A is selected, the current blockis coded using the motion vector MvL0_A having the prediction direction0 and the reference picture index RefL0_A. Then, only the merging blockcandidate index having a value of 0 which indicates use of theneighboring block A as shown in FIG. 4 is attached to a bitstream. Theamount of information on a prediction direction, a motion vector, and areference picture index is thereby reduced.

Furthermore, in the merging mode, a candidate which cannot be used forcoding (hereinafter referred to as an unusable-for-merging candidate),and a candidate having a combination of a prediction direction, a motionvector, and a reference picture index identical to a combination of aprediction direction, a motion vector, and a reference picture index ofany other merging block (hereinafter referred to as an identicalcandidate) are removed from merging block candidates as shown in FIG. 4.

In this manner, the total number of merging block candidates is reducedso that the amount of code assigned to merging block candidate indexescan be reduced. Here, “unusable for merging” means (1) that the mergingblock candidate has been coded by intra prediction, (2) that the mergingblock candidate is outside the boundary of a slice including the currentblock or the boundary of a picture including the current block, or (3)that the merging block candidate is yet to be coded.

In the example shown in FIG. 4, the neighboring block C is a block codedby intra prediction. The merging block candidate having the mergingblock candidate index 3 is therefore an unusable-for-merging candidateand removed from the merging block candidate list. The neighboring blockD is identical in prediction direction, motion vector, and referencepicture index to the neighboring block A. The merging block candidatehaving the merging block candidate index 4 is therefore removed from themerging block candidate list. As a result, the total number of themerging block candidates is finally three, and the size of the mergingblock candidate list is set at three.

Merging block candidate indexes are coded by variable-length coding byassigning bit sequences according to the size of each merging blockcandidate list as shown in FIG. 5. Thus, in the merging mode, the amountof code is reduced by changing bit sequences assigned to merging modeindexes according to the size of each merging block candidate list.

FIG. 6 is a flowchart showing an example of a process for coding whenthe merging mode is used. In Step S1001, motion vectors, referencepicture indexes, and prediction directions of merging block candidatesare obtained from neighboring blocks and a co-located block. In StepS1002, identical candidates and unusable-for-merging candidates areremoved from the merging block candidates. In Step S1003, the totalnumber of the merging block candidates after the removing is set as thesize of the merging block candidate list. In Step S1004, the mergingblock candidate index to be used in coding the current block isdetermined. In Step S1005, the determined merging block candidate indexis coded by performing variable-length coding in bit sequence accordingto the size of the merging block candidate list.

FIG. 7 is a flowchart showing an example of a process for decoding usingthe merging mode. In Step S2001, motion vectors, reference pictureindexes, and prediction directions of merging block candidates areobtained from neighboring blocks and a co-located block. In Step S2002,identical candidates and unusable-for-merging candidates are removedfrom the merging block candidates. In Step S2003, the total number ofthe merging block candidates after the removing is set as the size ofthe merging block candidate list. In Step S2004, the merging blockcandidate index to be used in decoding a current block is decoded from abitstream using the size of the merging block candidate list. In StepS2005, decoding of a current block is performed by generating aprediction picture using the merging block candidate indicated by thedecoded merging block candidate index.

FIG. 8 shows syntax for attachment of merging block candidate indexes toa bitstream. In FIG. 8, merge_idx represents a merging block candidateindex, and merge_flag represents a merging flag. NumMergeCand representsthe size of a merging block candidate list. NumMergeCand is set at thetotal number of merging block candidates after unusable-for-mergingcandidates and identical candidates are removed from the merging blockcandidates.

Coding or decoding of an image is performed using the merging mode inthe above-described manner.

However, in the above-described merging mode, whether to useuni-directional prediction or bi-directional prediction for coding acurrent block depends on whether a merging block candidate selected isuni-directionally predicted or bi-directionally predicted. Accordingly,for example, when all merging block candidates are coded usinguni-directional prediction, only uni-directional prediction can be usedfor coding a current block in the merging mode. In other words, eventhough a current block would be coded more efficiently usingbi-directional prediction than uni-directional prediction, onlyuni-directional prediction can be used for the coding of the currentblock. This may result in decrease in coding efficiency.

In view of this, an image coding method according to an aspect of thepresent invention is a method for coding an image on a block-by-blockbasis to generate a bitstream, and includes: determining a maximumnumber of merging candidates each of which is a combination of aprediction direction, a motion vector, and a reference picture index foruse in coding of a current block; deriving a plurality of first mergingcandidates based on prediction directions, motion vectors, and referencepicture indexes used in coding of blocks spatially or temporallyneighboring the current block; determining whether or not a total numberof the derived first merging candidates is smaller than the maximumnumber; deriving, by making a combination out of the derived firstmerging candidates, a second merging candidate for bi-directionalprediction when it is determined that the total number of the derivedfirst merging candidates is smaller than the maximum number; selecting amerging candidate to be used for the coding of the current block fromthe derived first merging candidates and the derived second mergingcandidate; and coding, using the determined maximum number, an index foridentifying the selected merging candidate, and attaching the codedindex to the bitstream.

With this, a second merging candidate for bi-directional prediction canbe derived by making a combination out of first merging candidatesderived based on blocks spatially or temporally neighboring a currentblock to be coded. In particular, a second merging candidate forbi-directional prediction can be derived even when the first mergingcandidates include no bi-directionally predicted merging candidate. As aresult, the variety of combinations of a prediction direction, a motionvector, and a reference picture index which are selected as mergingcandidates is increased so that coding efficiency can be increased.

Furthermore, a second merging candidate can be derived when it isdetermined that the total number of the first merging candidates issmaller than the maximum number. Accordingly, the total number ofmerging candidates can be increased within a range not exceeding themaximum number so that coding efficiency can be increased.

Furthermore, an index for identifying a merging candidate can be codedusing the determined maximum number. In other words, an index can becoded independently of the total number of actually derived mergingcandidates. Therefore, even when information necessary for derivation ofa merging candidate (for example, information on a co-located block) islost, an index can be still decoded and error resistance is therebyenhanced. Furthermore, an index can be decoded independently of thetotal number of actually derived merging candidates. In other words, anindex can be decoded without waiting for derivation of mergingcandidates. In other words, a bitstream can be generated for whichderiving of merging candidates and decoding of indexes can be performedin parallel.

For example, in the deriving of a second merging candidate, the secondmerging candidate may be derived by combining a motion vector and areference picture index for a first prediction direction which areincluded in one of the first merging candidates, and a motion vector anda reference picture index for a second prediction direction which areincluded in a different one of the first merging candidates.

With this, a second merging candidate for bi-directional prediction canbe derived by combining motion vectors and reference picture indexesincluded in two first merging candidates, where the motion vectors havedifferent prediction directions, and the reference picture indexes arefor different prediction directions.

For example, in the deriving of a plurality of first merging candidates,the plurality of first merging candidates may be derived such that eachof the first merging candidates is a different combination of aprediction direction, a motion vector, and a reference picture index.

With this, each of the derived first merging candidates is a differentcombination of a prediction direction, a motion vector, and a referencepicture index. As a result, the total number of the second mergingcandidates can be increased so that the variety of combinations of aprediction direction, a motion vector, and a reference picture indexwhich are selected as merging candidates can be increased. It istherefore possible to further increase coding efficiency.

For example, in the deriving of a plurality of first merging candidate,a combination of a prediction direction, a motion vector, and areference picture index may be derived as one of the plurality of firstmerging candidates, and the combination of the prediction direction,motion vector, and reference picture index has been used in coding ablock among blocks spatially neighboring the current block except ablock coded by intra prediction, a block outside a boundary of a sliceincluding the current block or a boundary of a picture including thecurrent block, and a block yet to be coded.

With this, a first merging candidate can be derived from blocksappropriate for obtainment of a merging candidate.

For example, in the coding, information indicating the determinedmaximum number may be further attached to the bitstream.

With this, information indicating the determined maximum number can beattached to a bitstream. It is therefore possible to switch maximumnumbers by the appropriate unit so that coding efficiency can beincreased.

For example, the image coding method may further include: switching acoding process between a first coding process conforming to a firststandard and a second coding process conforming to a second standard;and attaching, to the bitstream, identification information indicatingeither the first standard or the second standard to which the codingprocess after the switching conforms, wherein when the coding process isswitched to the first coding process, the determining of a maximumnumber of merging candidates, the deriving of a plurality of firstmerging candidates, the determining of whether or not the total numberof the derived plurality of first merging candidates is smaller than themaximum number, the deriving of a second merging candidate, theselecting, and the coding are performed as the first coding process.

With this, it is possible to switchably perform the first coding processconforming to the first standard and the second coding processconforming to the second standard.

Furthermore, an image decoding method according to an aspect of thepresent invention is a method for decoding, on a block-by-block basis, acoded image included in a bitstream, and includes: determining a maximumnumber of merging candidates each of which is a combination of aprediction direction, a motion vector, and a reference picture index foruse in decoding of a current block; deriving a plurality of firstmerging candidates based on prediction directions, motion vectors, andreference picture indexes used in decoding of blocks spatially ortemporally neighboring the current block; determining whether or not atotal number of the derived first merging candidates is smaller than themaximum number; deriving, by making a combination out of the derivedfirst merging candidates, a second merging candidate for bi-directionalprediction when it is determined that the total number of the derivedfirst merging candidates is smaller than the maximum number; decoding anindex coded and attached to the bitstream, using the determined maximumnumber, the index being an index for identifying a merging candidate;and selecting, based on the decoded index, a merging candidate to beused for the decoding of a current block, the merging candidate beingselected from the derived first merging candidates and the derivedsecond merging candidate.

With this, a second merging candidate for bi-directional prediction canbe derived by making a combination out of first merging candidatesderived based on blocks spatially or temporally neighboring a currentblock to be decoded. In particular, a new second merging candidate forbi-directional prediction can be derived even when the first mergingcandidates include no bi-directionally predicted merging candidate. As aresult, the variety of combinations of a prediction direction, a motionvector, and a reference picture index which can be selected as mergingcandidates is increased so that a bitstream coded with increasedefficiency can be appropriately decoded.

Furthermore, a second merging candidate can be derived when it isdetermined that the total number of the first merging candidates issmaller than the maximum number. Accordingly, the total number ofmerging candidates can be increased within a range not exceeding themaximum number so that a bitstream coded with further increased codingefficiency can be appropriately decoded.

Furthermore, an index for identifying a merging candidate can be decodedusing the determined maximum number. In other words, an index can bedecoded independently of the total number of actually derived mergingcandidates. Therefore, even when information necessary for derivation ofa merging candidate (for example, information on a co-located block) islost, an index can be still decoded and error resistance is therebyenhanced. Furthermore, an index can be decoded independently of thetotal number of actually derived merging candidates. In other words, anindex can be decoded without waiting for derivation of mergingcandidates. In other words, deriving of merging candidates and decodingof indexes can be performed in parallel.

For example, in the deriving of a second merging candidate, the secondmerging candidate may be derived by combining a motion vector and areference picture index for a first prediction direction which areincluded in one of the first merging candidates, and a motion vector anda reference picture index for a second prediction direction which areincluded in a different one of the first merging candidates.

With this, a second merging candidate for bi-directional prediction canbe derived by combining motion vectors and reference picture indexesincluded in two first merging candidates. The motion vectors havedifferent prediction directions, and the reference picture indexes arefor different prediction directions.

For example, in the deriving of a plurality of first merging candidates,the plurality of first merging candidates may be derived such that eachof the first merging candidates is a different combination of aprediction direction, a motion vector, and a reference picture index.

With this, first merging candidates are derived such that each of thefirst merging candidates is a different combination of a predictiondirection, a motion vector, and a reference picture index. As a result,the total number of the second merging candidates can be increased sothat the variety of combinations of a prediction direction, a motionvector, and a reference picture index for a selectable merging candidatecan be increased. It is therefore possible to appropriately decode abitstream coded with further increased coding efficiency.

For example, in the deriving of a plurality of first merging candidate,a combination of a prediction direction, a motion vector, and areference picture index may be derived as one of the plurality of firstmerging candidates, and the combination of the prediction direction,motion vector, and reference picture index has been used in decoding ofa block among blocks spatially neighboring the current block except ablock decoded by intra prediction, a block outside a boundary of a sliceincluding the current block or a boundary of a picture including thecurrent block, and a block yet to be decoded.

With this, a first merging candidate can be derived from blocksappropriate for obtainment of a merging candidate.

For example, in the determining of a maximum number of mergingcandidates, the maximum number may be determined based on informationattached to the bitstream and indicating the maximum number.

With this, a maximum number can be determined based on informationattached to a bitstream. It is therefore possible to decode an imagecoded using maximum numbers changed by the appropriate unit.

For example, the image decoding method may further include: switching adecoding process between a first decoding process conforming to a firststandard and a second decoding process conforming to a second standard,according to identification information attached to the bitstream andindicating either the first standard or the second standard, whereinwhen the decoding process is switched to the first decoding process, thedetermining of a maximum number of merging candidates, the deriving of aplurality of first merging candidates, the determining of whether or notthe total number of the derived plurality of first merging candidates issmaller than the maximum number, the deriving of a second mergingcandidate, the decoding, and the selecting are performed as the firstdecoding process.

With this, it is possible to switchably perform the first decodingprocess conforming to the first standard and the second decoding processconforming to the second standard.

It should be noted that these general or specific aspects can beimplemented as a system, a method, an integrated circuit, a computerprogram, a computer-readable recording medium such as a compact discread-only memory (CD-ROM), or as any combination of a system, a method,an integrated circuit, a computer program, and a computer-readablerecording medium.

An image coding apparatus and an image decoding apparatus according toan aspect of the present invention will be specifically described belowwith reference to the drawings.

Each of the exemplary embodiments described below shows a specificexample of the present invention. The numerical values, shapes,materials, constituent elements, the arrangement and connection of theconstituent elements, steps, the processing order of the steps etc.shown in the following exemplary embodiments are mere examples, andtherefore do not limit the present invention. Furthermore, among theconstituent elements in the following exemplary embodiments, constituentelements not recited in any one of the independent claims defining themost generic part of the inventive concept are not necessarily requiredin order to overcome the disadvantages.

The term “uni-directional prediction” as used herein refers toprediction with reference to only one of a first reference picture list(a reference picture list 0) and a second reference picture list (areference picture list 1). The term “bi-directional prediction” as usedherein refers to prediction with reference to both of the firstreference picture list and the second reference picture list.

It should be noted that bi-directional prediction need not be performedwith reference to a forward reference picture and a backward referencepicture. In other words, bi-directional prediction may be performed withreference to two reference pictures in the same direction (forward orbackward).

Embodiment 1

FIG. 9 is a block diagram showing a configuration of an image codingapparatus according to Embodiment 1. An image coding apparatus 100 codesan image on a block-by-block basis to generate a bitstream.

As shown in FIG. 9, the image coding apparatus 100 includes a subtractor101, an orthogonal transformation unit 102, a quantization unit 103, aninverse-quantization unit 104, an inverse-orthogonal-transformation unit105, an adder 106, block memory 107, frame memory 108, an intraprediction unit 109, an inter prediction unit 110, an inter predictioncontrol unit 111, a picture-type determination unit 112, a switch 113, amerging block candidate calculation unit 114, colPic memory 115, and avariable-length-coding unit 116.

The subtractor 101 subtracts, on a block-by-block basis, predictionpicture data from input image data included in an input image sequenceto generate prediction error data.

The orthogonal transformation unit 102 transforms the generatedprediction error data from a picture domain into a frequency domain.

The quantization unit 103 quantizes the prediction error datatransformed into a frequency domain.

The inverse-quantization unit 104 inverse-quantizes the prediction errordata quantized by the quantization unit 103.

The inverse-orthogonal-transformation unit 105 transforms theinverse-quantized prediction error data from a frequency domain into apicture domain.

The adder 106 adds, on a block-by-block basis, prediction picture dataand the prediction error data inverse-quantized by theinverse-orthogonal-transformation unit 105 to generate reconstructedimage data.

The block memory 107 stores the reconstructed image data in units of ablock.

The frame memory 108 stores the reconstructed image data in units of aframe.

The picture-type determination unit 112 determines in which of thepicture types of I picture, B picture, and P picture the input imagedata is to be coded. Then, the picture-type determination unit 112generates picture-type information indicating the determined picturetype.

The intra prediction unit 109 generates intra prediction picture data ofa current block by performing intra prediction using reconstructed imagedata stored in the block memory 107 in units of a block.

The inter prediction unit 110 generates inter prediction picture data ofa current block by performing inter prediction using reconstructed imagedata stored in the frame memory 108 in units of a frame and a motionvector derived by a process including motion estimation.

When a current block is coded by intra prediction coding, the switch 113outputs intra prediction picture data generated by the intra predictionunit 109 as prediction picture data of the current block to thesubtractor 101 and the adder 106. On the other hand, when a currentblock is coded by inter prediction coding, the switch 113 outputs interprediction picture data generated by the inter prediction unit 110 asprediction picture data of the current block to the subtractor 101 andthe adder 106.

The merging block candidate calculation unit 114 derives merging blockcandidates using motion vectors and others of neighboring blocks of thecurrent block and a motion vector and others of the co-located block(colPic information) stored in the colPic memory 115. Furthermore, themerging block candidate calculation unit 114 adds the derived mergingblock candidates to a merging block candidate list.

Furthermore, the merging block candidate calculation unit 114 derives,as a new candidate, a merging block candidate (hereinafter referred toas a combined merging block candidate) by combining, using a methoddescribed later, (i) a motion vector and a reference picture index for aprediction direction 0 of one of the derived merging block candidatesand (ii) a motion vector a reference picture index for a predictiondirection 1 of a different one of the derived merging block candidates.Then, the merging block candidate calculation unit 114 adds the derivedcombined merging block candidate as a new merging block candidate to themerging block candidate list. Furthermore, the merging block candidatecalculation unit 114 calculates the total number of the merging blockcandidates.

Furthermore, the merging block candidate calculation unit 114 assignsmerging block candidate indexes each having a different value to thederived merging block candidates. Then, the merging block candidatecalculation unit 114 transmits the merging block candidates and mergingblock candidate indexes to the inter prediction control unit 111.Furthermore, the merging block candidate calculation unit 114 transmitsthe calculated total number of the merging block candidates to thevariable-length-coding unit 116.

The inter prediction control unit 111 selects a prediction mode usingwhich prediction error is the smaller from a prediction mode in which amotion vector derived by motion estimation is used (motion estimationmode) and a prediction mode in which a motion vector derived from amerging block candidate is used (merging mode). The inter predictioncontrol unit 111 also transmits a merging flag indicating whether or notthe selected prediction mode is the merging mode to thevariable-length-coding unit 116. Furthermore, the inter predictioncontrol unit 111 transmits a merging block candidate index correspondingto the determined merging block candidates to the variable-length-codingunit 116 when the selected prediction mode is the merging mode.Furthermore, the inter prediction control unit 111 transfers the colPicinformation including the motion vector and others of the current blockto the colPic memory 115.

The variable-length-coding unit 116 generates a bitstream by performingvariable-length coding on the quantized prediction error data, themerging flag, and the picture-type information. Thevariable-length-coding unit 116 also sets the total number of mergingblock candidates as the size of the merging block candidate list.Furthermore, the variable-length-coding unit 116 performsvariable-length coding on a merging block candidate index to be used incoding, by assigning, according to the size of the merging blockcandidate list, a bit sequence to the merging block candidate index.

FIG. 10 is a flowchart showing processing operations of the image codingapparatus 100 according to Embodiment 1.

In Step S101, the merging block candidate calculation unit 114 derivesmerging block candidates from neighboring blocks and a co-located blockof a current block. Furthermore, the merging block candidate calculationunit 114 calculates the size of a merging block candidate list using amethod described later.

For example, in the case shown in FIG. 3, the merging block candidatecalculation unit 114 selects the neighboring blocks A to D as mergingblock candidates. Furthermore, the merging block candidate calculationunit 114 calculates, as a merging block candidate, a co-located mergingblock having a motion vector, a reference picture index, and aprediction direction which are calculated from the motion vector of aco-located block using the time prediction mode.

The merging block candidate calculation unit 114 assigns merging blockcandidate indexes to the respective merging block candidates as shown in(a) in FIG. 11. Next, the merging block candidate calculation unit 114calculates a merging block candidate list and the size of the mergingblock candidate list as shown in (b) in FIG. 11 by removingunusable-for-merging candidates and identical candidates and adding newcombined merging block candidates using a method described later.

Shorter codes are assigned to merging block candidate indexes of smallervalues. In other words, the smaller the value of a merging blockcandidate index, the smaller the amount of information necessary forindicating the merging block candidate index.

On the other hand, the larger the value of a merging block candidateindex, the larger the amount of information necessary for the mergingblock candidate index. Therefore, coding efficiency will be increasedwhen merging block candidate indexes of smaller values are assigned tomerging block candidates which are more likely to have motion vectors ofhigher accuracy and reference picture indexes of higher accuracy.

Therefore, a possible case is that the merging block candidatecalculation unit 114 counts the total number of times of selection ofeach merging block candidates as a merging block, and assigns mergingblock candidate indexes of smaller values to blocks with a larger totalnumber of the times. Specifically, this can be achieved by specifying amerging block selected from neighboring blocks and assigning a mergingblock candidate index of a smaller value to the specified merging blockwhen a current block is coded.

When a merging block candidate does not have information such as amotion vector (for example, when the merging block has been a blockcoded by intra prediction, it is located outside the boundary of apicture or the boundary of a slice, or it is yet to be coded), themerging block candidate is unusable for coding.

In Embodiment 1, such a merging block candidate unusable for coding isreferred to as an unusable-for-merging candidate, and a merging blockcandidate usable for coding is referred to as a usable-for-mergingcandidate. In addition, among a plurality of merging block candidates, amerging block candidate identical in motion vector, reference pictureindex, and prediction direction to any other merging block is referredto as an identical candidate.

In the case shown in FIG. 3, the neighboring block C is anunusable-for-merging candidate because it is a block coded by intraprediction. The neighboring block D is an identical candidate because itis identical in motion vector, reference picture index, and predictiondirection to the neighboring block A.

In Step S102, the inter prediction control unit 111 selects a predictionmode based on comparison, using a method described later, betweenprediction error of a prediction picture generated using a motion vectorderived by motion estimation and prediction error of a predictionpicture generated using a motion vector obtained from a merging blockcandidate. When the selected prediction mode is the merging mode, theinter prediction control unit 111 sets the merging flag to 1, and whennot, the inter prediction control unit 111 sets the merging flag to 0.

In Step S103, whether or not the merging flag is 1 (that is, whether ornot the selected prediction mode is the merging mode) is determined.

When the result of the determination in Step S103 is true (Yes, S103),the variable-length-coding unit 116 attaches the merging flag to abitstream in Step S104. Subsequently, in Step S105, thevariable-length-coding unit 116 assigns bit sequences according to thesize of the merging block candidate list as shown in FIG. 5 to themerging block candidate indexes of merging block candidates to be usedfor coding. Then, the variable-length-coding unit 116 performsvariable-length coding on the assigned bit sequence.

On the other hand, when the result of the determination in Step S103 isfalse (S103, No), the variable-length-coding unit 116 attachesinformation on a merging flag and a motion estimation vector mode to abitstream in Step S106.

In Embodiment 1, a merging block candidate index having a value of “0”is assigned to the neighboring block A as shown in (a) in FIG. 11. Amerging block candidate index having a value of “1” is assigned to theneighboring block B. A merging block candidate index having a value of“2” is assigned to the co-located merging block. A merging blockcandidate index having a value of “3” is assigned to the neighboringblock C. A merging block candidate index having a value of “4” isassigned to the neighboring block D.

It should be noted that the merging block candidate indexes having sucha value may be assigned otherwise. For example, when a new combinedmerging block candidate is added using a method described later, thevariable-length-coding unit 116 may assign smaller values to preexistentmerging block candidates and a larger value to the new combined mergingblock candidate. In other words, the variable-length-coding unit 116 mayassign a merging block candidate index of a smaller value to apreexistent merging block candidate in priority to a new candidate.

Furthermore, merging block candidates are not limited to the blocks atthe positions of the neighboring blocks A, B, C, and D. For example, aneighboring block located above the lower left neighboring block D canbe used as a merging block candidate. Furthermore, it is not necessaryto use all the neighboring blocks as merging block candidates. Forexample, it is also possible to use only the neighboring blocks A and Bas merging block candidates.

Furthermore, although the variable-length-coding unit 116 attaches amerging block candidate index to a bitstream in Step S105 in FIG. 10 inEmbodiment 1, attaching such a merging block candidate index to abitstream is not always necessary. For example, thevariable-length-coding unit 116 need not attach a merging blockcandidate index to a bitstream when the size of the merging blockcandidate list is “1”. The amount of information on the merging blockcandidate index is thereby reduced.

FIG. 12 is a flowchart showing details of the process in Step S101 inFIG. 10. Specifically, FIG. 12 illustrates a method of calculatingmerging block candidates and the size of a merging block candidate list.FIG. 12 will be described below.

In Step S111, the merging block candidate calculation unit 114determines whether or not a merging block candidate [N] is ausable-for-merging candidate using a method described later.

Here, N denotes an index value for identifying each merging blockcandidate. In Embodiment 1, N takes values from 0 to 4. Specifically,the neighboring block A in FIG. 3 is assigned to a merging blockcandidate [0]. The neighboring block B in FIG. 3 is assigned to amerging block candidate [1]. The co-located merging block is assigned toa merging block candidate [2]. The neighboring block C in FIG. 3 isassigned to a merging block candidate [3]. The neighboring block D inFIG. 3 is assigned to a merging block candidate [4].

In Step S112, the merging block candidate calculation unit 114 obtainsthe motion vector, reference picture index, and prediction direction ofthe merging block candidate [N], and adds them to a merging blockcandidate list.

In Step S113, the merging block candidate calculation unit 114 searchesthe merging block candidate list for an unusable-for-merging candidateand an identical candidate, and removes the unusable-for-mergingcandidate and the identical candidate from the merging block candidatelist as shown in FIG. 11.

In Step S114, the merging block candidate calculation unit 114 adds anew combined merging block candidate to the merging block candidate listusing a method described later. Here, when a new combined merging blockcandidate is added, the merging block candidate calculation unit 114 mayreassign merging block candidate indexes so that the merging blockcandidate indexes of smaller values are assigned to preexistent mergingblock candidates in priority to the new candidate. In other words, themerging block candidate calculation unit 114 may reassign the mergingblock candidate indexes so that a merging block candidate index of alarger value is assigned to the new combined merging block candidate.The amount of code of merging block candidate indexes is therebyreduced.

In Step S115, the merging block candidate calculation unit 114 sets thetotal number of merging block candidates after the adding of thecombined merging block candidate as the size of the merging blockcandidate list. In the example shown in FIG. 11, the total number ofmerging block candidates is calculated to be “5”, and the size of themerging block candidate list is set at “5”.

The new combined merging block candidate in Step S114 is a candidatenewly added to merging block candidates using a method described laterwhen the total number of merging block candidates is smaller than amaximum number of merging block candidates. In this manner, when thetotal number of merging block candidates is smaller than a maximumnumber of merging block candidate, the image coding apparatus 100 adds acombined merging block candidate for bi-directional prediction so thatcoding efficiency can be increased.

FIG. 13 is a flowchart showing details of the process in Step S111 inFIG. 12. Specifically, FIG. 13 illustrates a method of determiningwhether or not a merging block candidate [N] is a usable-for-mergingcandidate and updating the total number of usable-for-mergingcandidates. FIG. 13 will be described below.

In Step S121, the merging block candidate calculation unit 114determines whether it is true or false that (1) a merging blockcandidate [N] has been coded by intra prediction, (2) the merging blockcandidate [N] is a block outside the boundary of a slice including thecurrent block or the boundary of a picture including the current block,or (3) the merging block candidate [N] is yet to be coded.

When the result of the determination in Step 121 is true (S121, Yes),the merging block candidate calculation unit 114 sets the merging blockcandidate [N] as an unusable-for-merging candidate in Step S122. On theother hand, when the result of the determination in Step S121 is false(S121, No), the merging block candidate calculation unit 114 sets themerging block candidate [N] as a usable-for-merging candidate in StepS123.

FIG. 14 is a flowchart showing details of the process in Step S114 inFIG. 12. Specifically, FIG. 14 illustrates a method of adding a combinedmerging block candidate. FIG. 14 will be described below.

In Step S131, the merging block candidate calculation unit 114determines whether or not the total number of merging block candidatesis smaller than the total number of usable-for-merging candidates. Inother words, the merging block candidate calculation unit 114 determineswhether or not the total number of merging block candidates is stillbelow the maximum number of merging block candidates.

Here, when the result of the determination in Step S131 is true (S131,Yes), in Step S132, the merging block candidate calculation unit 114determines whether or not there is a new combined merging blockcandidate which can be added as a merging block candidate to the mergingblock candidate list. Here, when the result of the determination in StepS132 is true (S132, Yes), in Step S133, the merging block candidatecalculation unit 114 assigns a merging block candidate index to the newcombined merging block candidate and adds the new combined merging blockcandidate to the merging block candidate list. Furthermore, in StepS134, the merging block candidate calculation unit 114 increments thetotal number of merging block candidates by one. When the result of thedetermination in Step S132 is false (S132, No), the process returns toStep S131 to calculate a next combined merging block candidate.

On the other hand, when the result of the determination in Step S131 isfalse (S131 or S132, No), the process for adding a combined mergingblock candidate ends. In other words, the process for adding a newcombined merging block candidate is ended when the total number ofmerging block candidates reaches the maximum number of merging blockcandidates.

FIG. 15 is a flowchart showing details of the process in Step S132 inFIG. 14. Specifically, FIG. 15 illustrates a method of determiningwhether or not there is a combined merging block candidate. FIG. 15 willbe described below.

In Step S141, the merging block candidate calculation unit 114 updatesthe merging block candidate indexes idx1 and idx2. The merging blockcandidate indexes idx1 and idx2 are assigned to merging block candidates[idx1] and [idx2], respectively, and used in determining whether or nota combined merging block candidate can be generated using motion vectorsand reference picture indexes of the merging block candidates [idx1] and[idx2].

For example, first, the merging block candidate calculation unit 114updates the merging block candidate indexes idx1 and idx2 to [0] and[1], respectively. Next, the merging block candidate calculation unit114 determines whether or not there is a combined merging blockcandidate by performing the process from Step S142 through Step S149 todetermine whether or not a combined merging block candidate can begenerated using motion vectors and reference picture indexes of themerging block candidate [0] and the merging block candidate [1] includedin a merging block candidate list.

Next, in order to determine whether or not there is another combinedmerging block candidate, the merging block candidate calculation unit114 updates the merging block candidate indexes idx1 and idx2 to, forexample, [0] and [2]. Next, the merging block candidate calculation unit114 determines whether or not there is a combined merging blockcandidate by performing the process from Step S142 through Step S149 todetermine whether or not a combined merging block candidate can begenerated using motion vectors and reference picture indexes of themerging block candidate [0] and the merging block candidate [2] includedin the merging block candidate list.

Next, in order to determine whether or not there is another combinedmerging block candidate, the merging block candidate calculation unit114 updates the merging block candidate indexes idx1 and idx2 to, forexample, [0] and [3].

In this manner, the merging block candidate calculation unit 114determines whether or not there is a combined merging block candidate byincrementally updating the merging block candidate indexes idx1 and idx2in Step S141 and performing the process from Step S142 through Step S149after making a determination as to whether or not there is anothercombined merging block candidate.

It should be noted that details of the process for updating the mergingblock candidate indexes idx1 and idx2 is not limited by theabove-described procedure. The process can be performed using anyprocedure as long as the determination of whether or not there is acombined merging block candidate can be made for all combinations of themerging block candidates previously derived.

Next, in Step S142, the merging block candidate calculation unit 114determines whether or not all the following are true: (1) the mergingblock candidate indexes idx1 and idx2 have different values; (2) themerging block candidate [idx1] is not a combined merging blockcandidate; and (3) the merging block candidate [idx2] is not a combinedmerging block candidate.

When the result of the determination in Step S142 is true (S142, Yes),in Step S143, the merging block candidate calculation unit 114determines whether it is true or false that the merging block candidate[idx1] and the merging block candidate [idx2] are either (1) differentin prediction directions or (2) both bi-directionally predicted. Whenthe result of the determination in Step S143 is true, (S143, Yes), inStep S144, the merging block candidate calculation unit 114 determineswhether both the following are true: (1) the merging block candidate[idx1] is predicted in a prediction direction 0 or bi-directionallypredicted; and (2) the merging block candidate [idx2] is predicted in aprediction direction 1 or bi-directionally predicted. In other words,the merging block candidate calculation unit 114 determines whether ornot the merging block candidate [idx1] has at least a motion vectorhaving a prediction direction 0, and the merging block candidate [idx2]has at least a motion vector having a prediction direction 1.

When the result of the determination in Step S144 is true (S144, Yes),in Step S145 a, the merging block candidate calculation unit 114 assignsthe motion vector and a reference picture index for the predictiondirection 0 of the merging block candidate [idx1] to the predictiondirection 0 of the combined merging block. Furthermore, in Step S146 a,the merging block candidate calculation unit 114 assigns the motionvector and a reference picture index for the prediction direction 1 ofthe merging block candidate [idx2] to the prediction direction 1 of thecombined merging block. The merging block candidate calculation unit 114thus calculates a combined merging block for bi-directional prediction.

On the other hand, when the result of the determination in Step S144 isfalse (S144, No), in Step S145 b, the merging block candidatecalculation unit 114 assigns a motion vector and a reference pictureindex for the prediction direction 0 of the merging block candidate[idx2] to the prediction direction 0 of the combined merging block.Furthermore, in Step S146 b, the merging block candidate calculationunit 114 assigns a motion vector and a reference picture index for theprediction direction 1 of the merging block candidate [idx1] to theprediction direction 1 of the combined merging block. The merging blockcandidate calculation unit 114 thus calculates a combined merging blockfor bi-directional prediction.

In Step S147, the merging block candidate calculation unit 114determines whether or not the merging block candidate list alreadyincludes a merging block candidate identical in motion vector, referencepicture index, and prediction direction to the calculated combinedmerging block candidate. In other words, the merging block candidatecalculation unit 114 determines whether or not the combined mergingblock is different from any other merging block candidate.

When the result of the determination in Step S147 is true (S147, Yes),in Step S148, the merging block candidate calculation unit 114determines that there is a combined merging block candidate.

When the result of the determination in Step S142, Step S143, or StepS147 is false (S142, S143, or S147, No), the merging block candidatecalculation unit 114 determines in Step S149 that there is no combinedmerging block candidate.

FIG. 16 is a flowchart showing details of the process in Step S102 inFIG. 10. Specifically, FIG. 16 illustrates a process for selecting amerging block candidate. FIG. 16 will be described below.

In Step S151, the inter prediction control unit 111 sets a merging blockcandidate index at 0, the minimum prediction error at the predictionerror (cost) in the motion vector estimation mode, and a merging flag at0. Here, the cost is calculated using the following formula for an R-Doptimization model, for example.Cost=D+λR  (Equation 1)

In Equation 1, D denotes coding distortion. For example, D is the sum ofabsolute differences between original pixel values of a current block tobe coded and pixel values obtained by coding and decoding of the currentblock using a prediction picture generated using a motion vector. Rdenotes the amount of generated codes. For example, R is the amount ofcode necessary for coding a motion vector used for generation of aprediction picture. λ denotes an undetermined Lagrange multiplier.

In Step S152, the inter prediction control unit 111 determines whetheror not the value of a merging block candidate index is smaller than thetotal number of merging block candidates of a current block. In otherwords, the inter prediction control unit 111 determines whether or notthere is still a merging block candidate on which the process from StepS153 to Step S155 has not been performed yet.

When the result of the determination in Step S152 is true (S152, Yes),in Step S153, the inter prediction control unit 111 calculates the costfor a merging block candidate to which a merging block candidate indexis assigned. Then, in Step S154, the inter prediction control unit 111determines whether or not the calculated cost for a merging blockcandidate is smaller than the minimum prediction error.

Here, when the result of the determination in Step S154 is true, (S154,Yes), the inter prediction control unit 111 updates the minimumprediction error, the merging block candidate index, and the value ofthe merging flag in Step S155. On the other hand, when the result of thedetermination in Step S154 is false (S154, No), the inter predictioncontrol unit 111 does not update the minimum prediction error, themerging block candidate index, or the value of the merging flag.

In Step S156, the inter prediction control unit 111 increments themerging block candidate index by one, and repeats from Step S152 to StepS156.

On the other hand, when the result of the determination in Step S152 isfalse (S152, No), that is, there is no more unprocessed merging blockcandidate, the inter prediction control unit 111 fixes the final valuesof the merging flag and merging block candidate index in Step S157.

In this manner, the image coding apparatus 100 according to Embodiment 1calculates a new merging block candidate for bi-directional predictionbased on merging block candidates already derived so that codingefficiency can be increased. More specifically, the image codingapparatus 100 is capable of calculating a new merging block candidatefor bi-directional prediction (a combined merging block candidate) basedon merging block candidates calculated from neighboring blocks andco-located blocks by combining a motion vector and a reference pictureindex for a prediction direction 0 of one of the merging blockcandidates and a motion vector and a reference picture index for aprediction direction 1 of a different one of the merging blockcandidates. The image coding apparatus 100 then adds the calculatedcombined merging block candidate to a merging block candidate list sothat coding efficiency can be increased. Furthermore, the image codingapparatus 100 removes unusable-for-merging candidates and identicalcandidates from a merging block candidate list, and then adds a combinedmerging block candidate to the merging block candidate list so thatcoding efficiency can be increased without increasing a maximum numberof merging block candidates.

It should be noted that the example described in Embodiment 1 in whichmerging flag is always attached to a bitstream in merging mode is notlimiting. For example, the merging mode may be forcibly selecteddepending on a block shape for use in inter prediction of a currentblock. In this case, the amount of information may be reduced byattaching no merging flag to a bitstream.

It should be noted that the example described in Embodiment 1 where themerging mode is used in which a current block is coded using aprediction direction, a motion vector, and a reference picture indexcopied from a neighboring block of the current block is not limiting.For example, a skip merging mode may be used. In the skip merging mode,a current block is coded in the same manner as in the merging mode,using a prediction direction, a motion vector, and a reference pictureindex copied from a neighboring block of the current block withreference to a merging block candidate list created as shown in (b) inFIG. 11. When all resultant prediction errors are zero for the currentblock, a skip flag set at 1 and the skip flag and a merging blockcandidate index are attached to a bitstream. When any of the resultantprediction errors is non-zero, a skip flag is set at 0 and the skipflag, a merging flag, a merging block candidate index, and data of theprediction errors are attached to a bitstream.

It should be noted that the example described in Embodiment 1 where themerging mode is used in which a current block is coded using aprediction direction, a motion vector, and a reference picture indexcopied from a neighboring block of the current block is not limiting.For example, a motion vector in the motion vector estimation mode may becoded using a merging block candidate list created as shown in (b) inFIG. 11. Specifically, a difference is calculated by subtracting amotion vector of a merging block candidate indicated by a merging blockcandidate index from a motion vector in the motion vector estimationmode. Then, the calculated difference and the merging block candidateindex may be attached to a bitstream.

Optionally, a difference may be calculated by scaling a motion vectorMV_Merge of a merging block candidate using a reference picture indexRefIdx_ME in the motion estimation mode and a reference picture indexRefIdx_Merge of the merging block candidate and subtracting a motionvector scaledMV_Merge of the merging block candidate after the scalingfrom the motion vector in the motion estimation mode. Furthermore, thecalculated difference and the merging block candidate index may beattached to a bitstream. The following is an exemplary formula for thescaling.scaledMV_Merge=MV_Merge×(POC(RefIdx_(—)ME)−curPOC)/(POC(RefIdx_Merge)−curPOC)  (Equation 2)

Here, POC (RefIdx_ME) denotes the display order of a reference pictureindicated by a reference picture index RefIdx_ME. POC (RefIdx_Merge)denotes the display order of a reference picture indicated by areference picture index RefIdx_Merge. curPOC denotes the display orderof a current picture to be coded.

Embodiment 2

Although the merging block candidate calculation unit 114 determines inStep S147 in FIG. 15 whether or not a combined merging block candidateis an identical candidate in Embodiment 1, this determination is notalways necessary. For example, the determination in Step S147 may beomitted. This reduces computational complexity in derivation of amerging block candidate list for the image coding apparatus 100.

Furthermore, it should be noted that Embodiment 1 in which combinedmerging block candidates are added to a merging block candidate listuntil the total number of merging block candidates reaches a maximumnumber of merging block candidates is not limiting. For example, themerging block candidate calculation unit 114 may determine in Step S131in FIG. 14 whether or not the total number of merging block candidateshas reached a predetermined threshold value which is smaller than amaximum number of merging block candidates. This reduces computationalcomplexity in derivation of a merging block candidate list for the imagecoding apparatus 100.

Furthermore, it should be noted that Embodiment 1 in which adding acombined merging block candidate to a merging block candidate list isended when the total number of merging block candidates reaches amaximum number of merging block candidates is not limiting. For example,the determination in Step S131 in FIG. 14 as to whether or not the totalnumber of merging block candidates has reached the maximum number ofmerging block candidates may be omitted, and the merging block candidatecalculation unit 114 may add all combined merging block candidates tothe merging block candidate list until it turns out that there is nomore new combined merging block candidate. This widens the range ofoptional merging block candidates for the image coding apparatus 100 sothat coding efficiency can be increased.

Such a modification of the image coding apparatus according toEmbodiment 1 will be specifically described below as an image codingapparatus according to Embodiment 2.

FIG. 17 is a block diagram showing a configuration of an image codingapparatus 200 according to Embodiment 2. The image coding apparatus 200codes an image on a block-by-block basis to generate a bitstream. Theimage coding apparatus 200 includes a merging candidate derivation unit210, a prediction control unit 220, and a coding unit 230.

The merging candidate derivation unit 210 corresponds to the mergingblock candidate calculation unit 114 in Embodiment 1. The mergingcandidate derivation unit 210 derives merging candidates. The mergingcandidate derivation unit 210 generates a merging candidate list inwhich, for example, indexes each identifying a different derived mergingcandidate (hereinafter referred to as merging candidate indexes) areassociated with the respective derived merging candidates.

The merging candidates are candidates each of which is a combination ofa prediction direction, a motion vector, and a reference picture indexfor use in coding of a current block. Specifically, each of the mergingcandidates is a combination including at least a set of a predictiondirection, a motion vector, and a reference picture index.

The merging candidates correspond to the merging block candidates inEmbodiment 1. The merging candidate list is the same as the mergingblock candidate list.

As shown in FIG. 17, the merging candidate derivation unit 210 includesa first derivation unit 211 and a second derivation unit 212.

The first derivation unit 211 derives first merging candidates based onprediction directions, motion vectors, and reference picture indexeswhich have been used in coding blocks spatially or temporallyneighboring the current block. Then, for example, the first derivationunit 211 registers first merging candidates derived in this manner inthe merging candidate list each in association with a different mergingcandidate index.

The spatially neighboring block is a block which is within a pictureincluding the current block and neighbors the current block.Specifically, the neighboring blocks A to D shown in FIG. 3 are examplesof the spatially neighboring block.

The temporally neighboring block is a block which is within a picturedifferent from a picture including the current block and corresponds tothe current block. Specifically, a co-located block is an example of thetemporally neighboring block.

It should be noted that the temporally neighboring block need not be ablock located in the same position as the current block (co-locatedblock). For example, the temporally neighboring block may be a blockneighboring the co-located block.

It should be noted that the first derivation unit 211 may derive, as afirst merging candidate, a combination of a prediction direction, amotion vector, and a reference picture index which have been used incoding blocks which spatially neighbor the current block exceptunusable-for-merging blocks. The unusable-for-merging block is a blockcoded by intra prediction, a block outside the boundary of a sliceincluding the current block or the boundary of a picture including thecurrent block, or a block yet to be coded. With this configuration, thefirst derivation unit 211 can derive first merging candidates fromblocks appropriate for obtaining merging candidates.

The second derivation unit 212 derives a second merging candidate forbi-directional prediction by making a combination out of the derivedfirst merging candidates. Specifically, the second derivation unit 212derives a second merging candidate for bi-directional prediction bycombining, for example, a motion vector and a reference picture indexfor a first prediction direction (prediction direction 0) which areincluded in one of the first merging candidates and a motion vector anda reference picture index for a second prediction direction (predictiondirection 1) which are included in a different one of the first mergingcandidates. More specifically, for example, the second derivation unit212 derives a second merging candidate in the same manner as thederiving of a combined merging block candidate in Embodiment 1 (see FIG.15, etc.).

Then, for example, the second derivation unit 212 registers secondmerging candidates derived in this manner in the merging candidate listeach in association with a different merging candidate index. At thistime, the second derivation unit 212 may register the second mergingcandidates in the merging candidate list so that the merging candidateindexes assigned to the first merging candidates are smaller than themerging candidate indexes assigned to the second merging candidates asin Embodiment 1. With this, the image coding apparatus 200 can reducethe code amount when the first merging candidates are more likely to beselected as a merging candidate to be used for coding than a secondmerging candidate so that coding efficiency can be increased.

The prediction control unit 220 selects a merging candidate to be usedfor coding a current block from the derived first merging candidates andsecond merging candidates. In other words, the prediction control unit220 selects a merging candidate to be used for coding a current blockfrom the merging candidate list.

The coding unit 230 attaches an index for identifying the selectedmerging candidate (merging candidate index) to a bitstream. For example,the coding unit 230 codes the merging candidate index using the sum ofthe total number of the derived first merging candidates and the totalnumber of derived second merging candidates, and attaches the codedmerging candidate index to a bitstream.

Next, operations of the image coding apparatus 200 in theabove-described configuration will be described below.

FIG. 18 is a flowchart showing processing operations of the image codingapparatus 200 according to Embodiment 2.

First, the first derivation unit 211 derives first merging candidates(S201). Subsequently, the second derivation unit 212 derives a secondmerging candidate (S202).

Next, the prediction control unit 220 selects a merging candidate to beused for coding a current block from the first merging candidates andsecond merging candidate (S203). For example, the prediction controlunit 220 selects a merging candidate for which the cost represented byEquation 1 is a minimum from the merging candidate list as in Embodiment1.

Finally, the coding unit 230 attaches an index for identifying theselected merging candidate to a bitstream (S204).

In this manner, the image coding apparatus 200 according to Embodiment 2is capable of deriving a second merging candidate for bi-directionalprediction by making a combination out of first merging candidatesderived based on blocks spatially or temporally neighboring a currentblock to be coded. In particular, the image coding apparatus 200 iscapable of deriving a second merging candidate for bi-directionalprediction even when the first merging candidates include nobi-directionally predicted merging candidate. As a result, the imagecoding apparatus 200 increases the variety of combinations of aprediction direction, a motion vector, and a reference picture indexfrom which a merging candidate is selected so that coding efficiency canbe increased.

Embodiment 3

FIG. 19 is a block diagram showing a configuration of an image decodingapparatus 300 according to Embodiment 3. The image decoding apparatus300 is an apparatus corresponding to the image coding apparatus 100according to Embodiment 1. Specifically, for example, the image decodingapparatus 300 decodes, on a block-by-block basis, coded images includedin a bitstream generated by the image coding apparatus 100 according toEmbodiment 1.

As shown in FIG. 19, the image decoding apparatus 300 includes avariable-length-decoding unit 301, an inverse-quantization unit 302, aninverse-orthogonal-transformation unit 303, an adder 304, block memory305, frame memory 306, an intra prediction unit 307, an inter predictionunit 308, an inter prediction control unit 309, a switch 310, a mergingblock candidate calculation unit 311, and colPic memory 312.

The variable-length-decoding unit 301 generates picture-typeinformation, a merging flag, and a quantized coefficient by performingvariable-length decoding on an input bitstream. Furthermore, thevariable-length-decoding unit 301 performs variable-length decoding on amerging block candidate index using the total number of merging blockcandidates calculated by the merging block candidate calculation unit311.

The inverse-quantization unit 302 inverse-quantizes the quantizedcoefficient obtained by the variable-length decoding.

The inverse-orthogonal-transformation unit 303 generates predictionerror data by transforming an orthogonal transformation coefficientobtained by the inverse quantization from a frequency domain to apicture domain.

The block memory 305 stores, in units of a block, decoded image datagenerated by adding the prediction error data and prediction picturedata.

The frame memory 306 stores decoded image data in units of a frame.

The intra prediction unit 307 generates prediction picture data of acurrent block to be decoded, by performing intra prediction using thedecoded image data stored in the block memory 305 in units of a block.

The inter prediction unit 308 generates prediction picture data of acurrent block to be decoded, by performing inter prediction using thedecoded image data stored in the frame memory 306 in units of a frame.

When a current block is decoded by intra prediction decoding, the switch310 outputs intra prediction picture data generated by the intraprediction unit 307 as prediction picture data of the current block tothe adder 304. On the other hand, when a current block is decoded byinter prediction decoding, the switch 310 outputs inter predictionpicture data generated by the inter prediction unit 308 as predictionpicture data of the current block to the adder 304.

The merging block candidate calculation unit 311 derives merging blockcandidates from motion vectors and others of neighboring blocks of thecurrent block and a motion vector and others of a co-located block(colPic information) stored in the colPic memory 312. Furthermore, themerging block candidate calculation unit 311 adds the derived mergingblock candidates to a merging block candidate list.

Furthermore, the merging block candidate calculation unit 311 derives,as a new candidate, a combined merging block candidate by combining,using a method described later, a motion vector and a reference pictureindex for a prediction direction 0 of one of the derived merging blockcandidates and a motion vector and a reference picture index for aprediction direction 1 of a different one of the derived merging blockcandidates. Then, the merging block candidate calculation unit 311 addsthe derived combined merging block candidate to the merging blockcandidate list. Furthermore, the merging block candidate calculationunit 311 calculates the total number of merging block candidates.

Furthermore, the merging block candidate calculation unit 311 assignsmerging block candidate indexes each having a different value to themerging block candidates. Then, the merging block candidate calculationunit 311 transmits the merging block candidates to which the mergingblock candidate indexes have been assigned to the inter predictioncontrol unit 309. Furthermore, the merging block candidate calculationunit 311 transmits the calculated total number of merging blockcandidates to the variable-length-decoding unit 301.

The inter prediction control unit 309 causes the inter prediction unit308 to generate an inter prediction picture using information on motionvector estimation mode when the merging flag decoded is “0”. On theother hand, when the merging flag is “1”, the inter prediction controlunit 309 determines, based on a decoded merging block candidate index, amotion vector, a reference picture index, and a prediction direction foruse in inter prediction from a plurality of merging block candidates.Then, the inter prediction control unit 309 causes the inter predictionunit 308 to generate an inter prediction picture using the determinedmotion vector, reference picture index, and prediction direction.Furthermore, the inter prediction control unit 309 transfers colPicinformation including the motion vector of the current block to thecolPic memory 312.

Finally, the adder 304 generates decoded image data by adding theprediction picture data and the prediction error data.

FIG. 20 is a flowchart showing processing operations of the imagedecoding apparatus 300 according to Embodiment 3.

In Step S301, the variable-length-decoding unit 301 decodes a mergingflag.

In Step S302, when the merging flag is “1” (S302, Yes), in Step S303,the merging block candidate calculation unit 311 generates a mergingblock candidate in the same manner as in Step S101 in FIG. 10.Furthermore, the merging block candidate calculation unit 311 calculatesthe total number of merging block candidates as the size of a mergingblock candidate list.

In Step S304, the variable-length-decoding unit 301 performsvariable-length decoding on a merging block candidate index from abitstream using the size of the merging block candidate list.

In Step S305, the inter prediction control unit 309 causes the interprediction unit 308 to generate an inter prediction picture using themotion vector, reference picture index, and prediction direction of themerging block candidate indicated by the decoded merging block candidateindex.

When the merging flag is “0” in Step S302 (S302, No), in Step S306, theinter prediction unit 308 generates an inter prediction picture usinginformation on motion vector estimation mode decoded by thevariable-length-decoding unit 301.

Optionally, when the size of a merging block candidate list calculatedin Step S303 is “1”, a merging block candidate index may be estimated tobe “0” without being decoded.

In this manner, the image decoding apparatus 300 according to Embodiment3 calculates a new merging block candidate for bi-directional predictionbased on merging block candidates already derived so that a bitstreamcode with increased coding efficiency can be appropriately decoded. Morespecifically, the image decoding apparatus 300 is capable of calculatinga new merging block candidate for bi-directional prediction (a combinedmerging block candidate) based on merging block candidates calculatedfrom neighboring blocks and co-located blocks by combining a motionvector and a reference picture index for a prediction direction 0 of oneof the merging block candidates and a motion vector and a referencepicture index for a prediction direction 1 of a different one of themerging block candidates. The image coding apparatus 100 then adds thecalculated combined merging block candidate to a merging block candidatelist so that a bitstream coded with increased efficiency can beappropriately decoded.

Furthermore, the image coding apparatus 100 removes unusable-for-mergingcandidates and identical candidates from a merging block candidate list,and then adds a combined merging block candidate to the merging blockcandidate list so that a bitstream coded with increased efficiency canbe appropriately decoded without increasing a maximum number of mergingblock candidates.

Embodiment 4

Although the image decoding apparatus according to Embodiment 3 includesconstituent elements as shown in FIG. 19, the image decoding apparatusneed not include all of the constituent elements. Such a modification ofthe image decoding apparatus according to Embodiment 3 will bespecifically described below as an image decoding apparatus according toEmbodiment 4.

FIG. 21 is a block diagram showing a configuration of an image decodingapparatus 400 according to Embodiment 4. The image decoding apparatus400 is an apparatus corresponding to the image coding apparatus 200according to Embodiment 2. Specifically, for example, the image decodingapparatus 400 decodes, on a block-by-block basis, coded images includedin a bitstream generated by the image coding apparatus 200 according toEmbodiment 2.

As shown in FIG. 21, the image decoding apparatus 400 includes a mergingcandidate derivation unit 410, a decoding unit 420, and a predictioncontrol unit 430.

The merging candidate derivation unit 410 corresponds to the mergingblock candidate calculation unit 311 in Embodiment 3. The mergingcandidate derivation unit 410 derives merging candidates. The mergingcandidate derivation unit 410 generates a merging candidate list inwhich, for example, indexes each identifying a different derived mergingcandidate (merging candidate indexes) are associated with the respectivederived merging candidates.

As shown in FIG. 21, the merging candidate derivation unit 410 includesa first derivation unit 411 and a second derivation unit 412.

The first derivation unit 411 derives first merging candidates in thesame manner as the first derivation unit 211 in Embodiment 2.Specifically, the first derivation unit 411 derives first mergingcandidates based on prediction directions, motion vectors, and referencepicture indexes which have been used for coding blocks spatially ortemporally neighboring a current block to be decoded. Then, for example,the first derivation unit 411 registers first merging candidates derivedin this manner in the merging candidate list each in association with adifferent merging candidate index.

The second derivation unit 412 derives a second merging candidate forbi-directional prediction by making a combination out of the derivedfirst merging candidates. Specifically, the second derivation unit 412derives a second merging candidate in the same manner as the secondderivation unit 212 in Embodiment 2. Then, for example, the secondderivation unit 412 registers second merging candidates derived in thismanner in the merging candidate list each in association with adifferent merging candidate index.

More specifically, the second derivation unit 412 derives a secondmerging candidate for bi-directional prediction by combining, forexample, a motion vector and a reference picture index for a firstprediction direction (prediction direction 0) which are included in oneof the first merging candidates and a motion vector and a referencepicture index for a second prediction direction (prediction direction 1)which are included in a different one of the first merging candidates.

The decoding unit 420 obtains an index for identifying a mergingcandidate from a bitstream. For example, the decoding unit 420 obtains amerging candidate index by decoding a coded merging candidate indexattached to a bitstream using the sum of the total number of the derivedfirst merging candidates and the total number of the derived secondmerging candidates (total number of merging candidates).

The prediction control unit 430 selects, based on the obtained index, amerging candidate to be used for decoding a current block from thederived first merging candidates and second merging candidates. In otherwords, the prediction control unit 430 selects a merging candidate to beused for decoding a current block from the merging candidate list.

Next, operations of the image decoding apparatus 400 in theabove-described configuration will be explained below.

FIG. 22 is a flowchart showing processing operations of the imagedecoding apparatus 400 according to Embodiment 4.

First, the first derivation unit 411 derives first merging candidates(S401). Subsequently, the second derivation unit 412 derives a secondmerging candidate (S402). Next, the decoding unit 420 obtains a mergingcandidate index from a bitstream (S403).

Finally, the prediction control unit 220 selects, based on the obtainedindex, a merging candidate to be used for decoding a current block fromthe first merging candidates and second merging candidate (S404).

In this manner, the image decoding apparatus 400 according to Embodiment4 is capable of deriving a second merging candidate for bi-directionalprediction by making a combination out of first merging candidatesderived based on blocks spatially or temporally neighboring a currentblock to be decoded. In particular, the image decoding apparatus 400 iscapable of deriving a second merging candidate for bi-directionalprediction even when the first merging candidates include nobi-directionally predicted merging candidate. As a result, the imagedecoding apparatus 400 increases the variety of combinations of aprediction direction, a motion vector, and a reference picture indexfrom which a merging candidate is selected so that a bitstream codedwith increased efficiency can be appropriately decoded.

Embodiment 5

Embodiment 5 is different in the method of deriving the size of amerging block candidate list from Embodiment 1. The method of derivingthe size of a merging block candidate list according to Embodiment 7will be described below in detail.

In the merging mode according to Embodiment 1, the total number ofmerging block candidates is set as the size of a merging block candidatelist for use in coding or decoding of a merging block candidate index.The total number of merging block candidates is determined afterunusable-for-merging candidates or identical candidates are removedbased on information on reference pictures including a co-located block.

A discrepancy in bit sequence assigned to a merging block candidateindex is therefore caused between an image coding apparatus and an imagedecoding apparatus in the case where there is a difference in the totalnumber of merging block candidates between the image coding apparatusand the image decoding apparatus. As a result, the image decodingapparatus cannot decode a bitstream correctly.

For example, when information on a reference picture referenced as aco-located block is lost due to packet loss in a transmission path, themotion vector or the reference picture index of the co-located blockbecomes unknown. Accordingly, the information on a merging blockcandidate to be generated from the co-located block becomes unknown. Insuch a case, it is no longer possible to correctly removeunusable-for-merging candidates or identical candidates from the mergingblock candidates in decoding. As a result, the image decoding apparatusfails to obtain the correct size of a merging block candidate list, andit is therefore impossible to normally decode a merging block candidateindex.

Thus, the image coding apparatus according to Embodiment 5 calculatesthe size of a merging block candidate list for use in coding or decodingof a merging block candidate index, using a method independent ofinformation on reference pictures including a co-located block. Theimage coding apparatus 100 thereby achieves enhanced error resistance.

FIG. 23 is a block diagram showing a configuration of an image codingapparatus 500 according to Embodiment 5. For FIG. 23, the constituentelements in common with FIG. 9 are denoted with the same referencesigns, and description thereof is omitted.

As shown in FIG. 23, the image coding apparatus 500 includes asubtractor 101, an orthogonal transformation unit 102, a quantizationunit 103, an inverse-quantization unit 104, aninverse-orthogonal-transformation unit 105, block memory 107, framememory 108, an intra prediction unit 109, an inter prediction unit 110,an inter prediction control unit 111, a picture-type determination unit112, a switch 113, a merging block candidate calculation unit 514,colPic memory 115, and a variable-length-coding unit 516.

The merging block candidate calculation unit 514 derives merging blockcandidates for merging mode using motion vectors and others ofneighboring blocks of the current block and a motion vector and othersof the co-located block (colPic information) stored in the colPic memory115. Then, the merging block candidate calculation unit 514 calculatesthe total number of usable-for-merging candidates using a methoddescribed later.

Furthermore, the merging block candidate calculation unit 514 assignsmerging block candidate indexes each having a different value to thederived merging block candidates. Then, the merging block candidatecalculation unit 514 transmits the merging block candidates and mergingblock candidate indexes to the inter prediction control unit 111.Furthermore, the merging block candidate calculation unit 514 transmitsthe calculated total number of usable-for-merging candidates to thevariable-length-coding unit 116.

The variable-length-coding unit 516 generates a bitstream by performingvariable-length coding on the quantized prediction error data, themerging flag, and the picture-type information. Thevariable-length-coding unit 516 also sets the total number ofusable-for-merging candidates as the size of the merging block candidatelist. Furthermore, the variable-length-coding unit 516 performsvariable-length coding on a merging block candidate index to be used incoding, by assigning, according to the size of the merging blockcandidate list, a bit sequence to the merging block candidate index.

FIG. 24 is a flowchart showing processing operations of the image codingapparatus 500 according to Embodiment 5. For FIG. 24, the steps incommon with FIG. 10 are denoted with the same reference signs, anddescription thereof is omitted as appropriate.

In Step S501, the merging block candidate calculation unit 514 derivesmerging block candidates from neighboring blocks and a co-located blockof a current block. Furthermore, the merging block candidate calculationunit 514 calculates the size of a merging block candidate list using amethod described later.

For example, in the case shown in FIG. 3, the merging block candidatecalculation unit 514 selects the neighboring blocks A to D as mergingblock candidates. Furthermore, the merging block candidate calculationunit 514 calculates, as a merging block candidate, a co-located mergingblock having a motion vector and others which are calculated from themotion vector of a co-located block using the time prediction mode.

The merging block candidate calculation unit 514 assigns merging blockcandidate indexes to the respective merging block candidates as shown in(a) in FIG. 25. Next, the merging block candidate calculation unit 514calculates a merging block candidate list as shown in (b) in FIG. 25 andthe size of the merging block candidate list by removingunusable-for-merging candidates and identical candidates and adding newcandidates using a method described later.

Shorter codes are assigned to merging block candidate indexes of smallervalues. In other words, the smaller the value of a merging blockcandidate index, the smaller the amount of information necessary forindicating the merging block candidate index.

On the other hand, the larger the value of a merging block candidateindex, the larger the amount of information necessary for the mergingblock candidate index. Therefore, coding efficiency will be increasedwhen merging block candidate indexes of smaller values are assigned tomerging block candidates which are more likely to have motion vectors ofhigher accuracy and reference picture indexes of higher accuracy.

Therefore, a possible case is that the merging block candidatecalculation unit 514 counts the total number of times of selection ofeach merging block candidates as a merging block, and assigns mergingblock candidate indexes of smaller values to blocks with a larger totalnumber of the times. Specifically, this can be achieved by specifying amerging block selected from neighboring blocks and assigning a mergingblock candidate index of a smaller value to the specified merging blockwhen a current block is coded.

When a merging block candidate does not have information such as amotion vector (for example, when the merging block has been a blockcoded by intra prediction, it is located outside the boundary of apicture or the boundary of a slice, or it is yet to be coded), themerging block candidate is unusable for coding.

In Embodiment 5, such a merging block candidate unusable for coding isreferred to as an unusable-for-merging candidate, and a merging blockcandidate usable for coding is referred to as a usable-for-mergingcandidate. In addition, among a plurality of merging block candidates, amerging block candidate identical in motion vector, reference pictureindex, and prediction direction to any other merging block is referredto as an identical candidate.

In the case shown in FIG. 3, the neighboring block C is anunusable-for-merging candidate because it is a block coded by intraprediction. The neighboring block D is an identical candidate because itis identical in motion vector, reference picture index, and predictiondirection to the neighboring block A.

In Step S102, the inter prediction control unit 111 selects a predictionmode based on comparison between prediction error of a predictionpicture generated using a motion vector derived by motion estimation andprediction error of a prediction picture generated using a motion vectorobtained from a merging block candidate. When the selected predictionmode is the merging mode, the inter prediction control unit 111 sets themerging flag to 1, and when not, the inter prediction control unit 111sets the merging flag to 0.

In Step S103, whether or not the merging flag is 1 (that is, whether ornot the selected prediction mode is the merging mode) is determined.

When the result of the determination in Step S103 is true (Yes, S103),the variable-length-coding unit 516 attaches the merging flag to abitstream in Step S104. Subsequently, in Step S505, thevariable-length-coding unit 516 assigns bit sequences according to thesize of the merging block candidate list as shown in FIG. 5 to themerging block candidate indexes of merging block candidates to be usedfor coding. Then, the variable-length-coding unit 516 performsvariable-length coding on the assigned bit sequence.

On the other hand, when the result of the determination in Step S103 isfalse (S103, No), the variable-length-coding unit 516 attachesinformation on a merging flag and a motion estimation vector mode to abitstream in Step S106.

In Embodiment 5, a merging block candidate index having a value of “0”is assigned to the neighboring block A as shown in (a) in FIG. 25. Amerging block candidate index having a value of “1” is assigned to theneighboring block B. A merging block candidate index having a value of“2” is assigned to the co-located merging block. A merging blockcandidate index having a value of “3” is assigned to the neighboringblock C. A merging block candidate index having a value of “4” isassigned to the neighboring block D.

It should be noted that the merging block candidate indexes having sucha value may be assigned otherwise. For example, when a new candidate isadded using the method described in Embodiment 1 or a method describedlater, the variable-length-coding unit 516 may assign smaller values topreexistent merging block candidates and a larger value to the newcandidate. In other words, the variable-length-coding unit 516 mayassign a merging block candidate index of a smaller value to apreexistent merging block candidate in priority to a new candidate.

Furthermore, merging block candidates are not limited to the blocks atthe positions of the neighboring blocks A, B, C, and D. For example, aneighboring block located above the lower left neighboring block D canbe used as a merging block candidate. Furthermore, it is not necessaryto use all the neighboring blocks as merging block candidates. Forexample, it is also possible to use only the neighboring blocks A and Bas merging block candidates.

Furthermore, although the variable-length-coding unit 516 attaches amerging block candidate index to a bitstream in Step S505 in FIG. 24 inEmbodiment 5, attaching such a merging block candidate index to abitstream is not always necessary. For example, thevariable-length-coding unit 116 need not attach a merging blockcandidate index to a bitstream when the size of the merging blockcandidate list is “1”. The amount of information on the merging blockcandidate index is thereby reduced.

FIG. 26 is a flowchart showing details of the process in Step S501 inFIG. 24. Specifically, FIG. 26 illustrates a method of calculatingmerging block candidates and the size of a merging block candidate list.FIG. 26 will be described below.

In Step S511, the merging block candidate calculation unit 514determines whether or not a merging block candidate [N] is ausable-for-merging candidate using a method described later. Then, themerging block candidate calculation unit 514 updates the total number ofusable-for-merging candidates according to the result of thedetermination.

Here, N denotes an index value for identifying each merging blockcandidate. In Embodiment 5, N takes values from 0 to 4. Specifically,the neighboring block A in FIG. 3 is assigned to a merging blockcandidate [0]. The neighboring block B in FIG. 3 is assigned to amerging block candidate [1]. The co-located merging block is assigned toa merging block candidate [2]. The neighboring block C in FIG. 3 isassigned to a merging block candidate [3]. The neighboring block D inFIG. 5 is assigned to a merging block candidate [4].

In Step S512, the merging block candidate calculation unit 514 obtainsthe motion vector, reference picture index, and prediction direction ofthe merging block candidate [N], and adds them to a merging blockcandidate list.

In Step S513, the merging block candidate calculation unit 514 searchesthe merging block candidate list for an unusable-for-merging candidateand an identical candidate, and removes the unusable-for-mergingcandidate and the identical candidate from the merging block candidatelist as shown in FIG. 25.

In Step S514, the merging block candidate calculation unit 514 adds anew candidate to the merging block candidate list using a methoddescribed in Embodiment 1 or a method described later. Here, when a newcandidate is added, the merging block candidate calculation unit 514 mayreassign merging block candidate indexes so that the merging blockcandidate indexes of smaller values are assigned to preexistent mergingblock candidates in priority to the new candidate. In other words, themerging block candidate calculation unit 514 may reassign the mergingblock candidate indexes so that a merging block candidate index of alarger value is assigned to the new candidate. The amount of code ofmerging block candidate indexes is thereby reduced.

In Step S515, the merging block candidate calculation unit 514 sets thetotal number of usable-for-merging candidates calculated in Step S511 asthe size of the merging block candidate list. In the example shown inFIG. 25, the calculated number of usable-for-merging candidates is “4”,and the size of the merging block candidate list is set at “4”.

The new candidate in Step S514 is a candidate newly added to mergingblock candidates using the method described in Embodiment 1 or a methoddescribed later when the total number of merging block candidates issmaller than the total number of usable-for-merging candidates. Forexample, the new candidate is a combined merging block candidate.Examples of such a new candidate include a neighboring block locatedabove the lower-left neighboring block D in FIG. 3, a blockcorresponding to any of the neighboring blocks A, B, C, and D for aco-located block. Furthermore, examples of such a new candidate furtherinclude a block having a motion vector, a reference picture index, aprediction direction, and the like which are statistically obtained forthe whole or a certain region of a reference picture. Thus, when thetotal number of merging block candidates is smaller than the totalnumber of usable-for-merging candidates, the merging block candidatecalculation unit 514 adds a new candidate having a new motion vector, anew reference picture index, and a new prediction direction so thatcoding efficiency can be increased.

FIG. 27 is a flowchart showing details of the process in Step S511 inFIG. 26. Specifically, FIG. 27 illustrates a method of determiningwhether or not a merging block candidate [N] is a usable-for-mergingcandidate and updating the total number of usable-for-mergingcandidates. FIG. 27 will be described below.

In Step S521, the merging block candidate calculation unit 514determines whether it is true or false that (1) a merging blockcandidate [N] has been coded by intra prediction, (2) the merging blockcandidate [N] is a block outside the boundary of a slice including thecurrent block or the boundary of a picture including the current block,or (3) the merging block candidate [N] is yet to be coded.

When the result of the determination in Step S21 is true (S521, Yes),the merging block candidate calculation unit 514 sets the merging blockcandidate [N] as an unusable-for-merging candidate in Step S522. On theother hand, when the result of the determination in Step S521 is false(S521, No), the merging block candidate calculation unit 514 sets themerging block candidate [N] as a usable-for-merging candidate in StepS523.

In Step S524, the merging block candidate calculation unit 514determines whether it is true or false that the merging block candidate[N] is either a usable-for-merging candidate or a co-located mergingblock candidate. Here, when the result of the determination in Step S524is true (S524, Yes), the merging block candidate calculation unit 514updates the total number of merging block candidates by incrementing itby one in Step S525. On the other hand, when the result of thedetermination in Step S524 is false (S524, No), the merging blockcandidate calculation unit 514 does not update the total number ofusable-for-merging candidates.

Thus, when a merging block candidate is a co-located merging block, themerging block candidate calculation unit 514 increments the total numberof usable-for-merging candidate by one regardless of whether theco-located block is a usable-for-merging candidate or anunusable-for-merging candidate. This prevents discrepancy of the numbersof usable-for-merging candidates between the image coding apparatus andthe image decoding apparatus even when information on a co-locatedmerging block is lost due to an incident such as packet loss.

The total number of usable-for-merging candidates is set as the size ofthe merging block candidate list in Step S515 shown in FIG. 26.Furthermore, the size of the merging block candidate list is used invariable-length coding of merging block candidate indexes in Step S505shown in FIG. 24. This makes it possible for the image coding apparatus500 to generate a bitstream which can be normally decoded so thatmerging block candidate indexes can be obtained even when information onreference picture including a co-located block is lost.

FIG. 28 is a flowchart showing details of the process in Step S514 inFIG. 26. Specifically, FIG. 28 illustrates a method of adding a newcandidate. FIG. 28 will be described below.

In Step S531, the merging block candidate calculation unit 514determines whether or not the total number of merging block candidatesis smaller than the total number of usable-for-merging candidates. Inother words, the merging block candidate calculation unit 514 determineswhether or not the total number of merging block candidate is stillbelow the total number of usable-for-merging candidates.

Here, when the result of the determination in Step S531 is true (S531,Yes), in Step S532, the merging block candidate calculation unit 514determines whether or not there is a new candidate which can be added asa merging block candidate to the merging block candidate list. Here,when the result of the determination in Step S532 is true (S532, Yes),in Step S533, the merging block candidate calculation unit 514 assigns amerging block candidate index to the new candidate and adds the newcandidate to the merging block candidate list. Furthermore, the mergingblock candidate calculation unit 514 increments the total number ofmerging block candidate by one in Step S534.

On the other hand, when the result of the determination in Step S101 orin Step S532 is false (S531 or S532, No), the process for adding a newcandidate ends. In other words, the process for adding a new candidateis ended when the total number of merging block candidates reaches thetotal number of usable-for-merging candidates or when there is no newcandidate.

Thus, the image coding apparatus 500 according to Embodiment 5 iscapable of calculating the size of a merging block candidate list foruse in coding or decoding of a merging block candidate index, using amethod independent of information on reference pictures including aco-located block. The image coding apparatus 500 thereby achievesenhanced error resistance.

More specifically, regardless of whether or not a co-located mergingblock is a usable-for-merging candidate, the image coding apparatus 500according to Embodiment 5 increments the total number ofusable-for-merging candidates by one each time a merging block candidateis determined as a co-located merging block. Then, the image codingapparatus 500 determines a bit sequence to be assigned to a mergingblock candidate index, using the total number of usable-for-mergingcandidates calculated in this manner. The image coding apparatus 500 isthus capable of generating a bitstream from which the merging blockcandidate index can be decoded normally even when information onreference pictures including a co-located block is lost.

Furthermore, when the total number of merging block candidates issmaller than the total number of usable-for-merging candidates, theimage coding apparatus 500 according to Embodiment 5 adds, as a mergingblock candidate, a new candidate having a new motion vector, a newreference picture index, and a new prediction direction so that codingefficiency can be increased.

It should be noted that the example described in Embodiment 5 in whichmerging flag is always attached to a bitstream in merging mode is notlimiting. For example, the merging mode may be forcibly selecteddepending on a block shape for use in inter prediction of a currentblock. In this case, the amount of information may be reduced byattaching no merging flag to a bitstream.

It should be noted that the example described in Embodiment 5 where themerging mode is used in which a current block is coded using aprediction direction, a motion vector, and a reference picture indexcopied from a neighboring block of the current block is not limiting.For example, a skip merging mode may be used. In the skip merging mode,a current block is coded in the same manner as in the merging mode,using a prediction direction, a motion vector, and a reference pictureindex copied from a neighboring block of the current block withreference to a merging block candidate list created as shown in (b) inFIG. 25. When all resultant prediction errors are zero for the currentblock, a skip flag set at 1 and the skip flag and a merging blockcandidate index are attached to a bitstream. When any of the resultantprediction errors is non-zero, a skip flag is set at 0 and the skipflag, a merging flag, a merging block candidate index, and data of theprediction errors are attached to a bitstream.

It should be noted that the example described in Embodiment 5 where themerging mode is used in which a current block is coded using aprediction direction, a motion vector, and a reference picture indexcopied from a neighboring block of the current block is not limiting.For example, a motion vector in the motion vector estimation mode may becoded using a merging block candidate list created as shown in (b) inFIG. 25. Specifically, a difference is calculated by subtracting amotion vector of a merging block candidate indicated by a merging blockcandidate index from a motion vector in the motion vector estimationmode. Furthermore, the calculated difference and the merging blockcandidate index may be attached to a bitstream.

Optionally, a difference may be calculated by scaling a motion vectorMV_Merge of a merging block candidate using a reference picture indexRefIdx_ME in the motion estimation mode and a reference picture indexRefIdx_Merge of the merging block candidate as represented by Equation2, and subtracting a motion vector scaledMV_Merge of the merging blockcandidate after the scaling from the motion vector in the motionestimation mode. Furthermore, the calculated difference and the mergingblock candidate index may be attached to a bitstream.

Embodiment 6

In Embodiment 5, the image coding apparatus determines a bit sequence tobe assigned to a merging block candidate index using the total number ofusable-for-merging candidates incremented by one each time a mergingblock candidate is determined as a co-located merging block, regardlessof whether or not a co-located merging block is a usable-for-mergingcandidate. Optionally, for example, the image coding apparatus maydetermine a bit sequence to be assigned to a merging block candidateindex using the total number of usable-for-merging candidates calculatedby incrementing by one for each merging block candidate regardless ofwhether or not the merging block candidate is a co-located merging blockin Step S524 in FIG. 27. In other words, the image coding apparatus mayassign a bit sequence to a merging block candidate index using the sizeof a merging block candidate list fixed at a maximum number N of thetotal number of merging block candidates. In other words, the imagecoding apparatus may code merging block candidate indexes using the sizeof a merging block candidate list fixed at a maximum value N of thetotal number of merging block candidates on the assumption that themerging block candidates are all usable-for-merging candidates.

For example, in the case shown in Embodiment 5, when the maximum value Nof the total number of merging block candidates is five (the neighboringblock A, neighboring block B, co-located merging block, neighboringblock C, and neighboring block D), the image coding apparatus may codethe merging block candidate indexes using the size of the merging blockcandidate list fixedly set at five. Furthermore, for example, when themaximum value N of the total number of merging block candidates is four(the neighboring block A, neighboring block B, neighboring block C, andneighboring block D), the image coding apparatus may code the mergingblock candidate indexes using the size of the merging block candidatelist fixedly set at four.

In this manner, the image coding apparatus may determine the size of amerging block candidate list based on the maximum value of the totalnumber of merging block candidates. It is therefore possible to generatea bitstream from which a variable-length-decoding unit of an imagedecoding apparatus can decode a merging block candidate index withoutreferencing information on a neighboring block or on a co-located block,so that computational complexity for the variable-length-decoding unitcan be reduced.

Such a modification of the image coding apparatus according toEmbodiment 5 will be specifically described below as an image codingapparatus according to Embodiment 6.

FIG. 29 is a block diagram showing a configuration of an image codingapparatus 600 according to Embodiment 6. The image coding apparatus 600codes an image on a block-by-block basis to generate a bitstream. Theimage coding apparatus 600 includes a merging candidate derivation unit610, a prediction control unit 620, and a coding unit 630.

The merging candidate derivation unit 610 corresponds to the mergingblock candidate calculation unit 514 in Embodiment 5. The mergingcandidate derivation unit 610 derives merging candidates. The mergingcandidate derivation unit 610 generates a merging candidate list inwhich, for example, indexes each identifying a different derived mergingcandidate are associated with the respective derived merging candidates.

As shown in FIG. 29, the merging candidate derivation unit 610 includesa first determination unit 611, a first derivation unit 612, aspecification unit 613, a second determination unit 614, and a secondderivation unit 615.

The first determination unit 611 determines a maximum number of mergingcandidates. In other words, the first determination unit 611 determinesa maximum value N of the total number of merging block candidates.

For example, the first determination unit 611 determines a maximumnumber of the merging candidates based on characteristics of the inputimage sequence (such as a sequence, a picture, a slice, or a block).Optionally, for example, the first determination unit 611 may determinea predetermined number as a maximum number of merging candidates.

More specifically, the first derivation unit 612 derives first mergingcandidates based on, for example, prediction directions, motion vectors,and reference picture indexes which have been used in coding blocksspatially or temporally neighboring the current block. Here, the firstderivation unit 612 derives first merging candidates within a range inwhich the total number of the first merging candidates does not exceedthe maximum number. Then, for example, the first derivation unit 612registers the first merging candidates derived in this manner in themerging candidate list each in association with a different mergingcandidate indexes.

It should be noted that the first derivation unit 612 may derive, as afirst merging candidate, a combination of a prediction direction, amotion vector, and a reference picture index which have been used incoding blocks which spatially neighbor the current block exceptunusable-for-merging blocks. An unusable-for-merging block is a blockcoded by intra prediction, a block outside the boundary of a sliceincluding the current block or the boundary of a picture including thecurrent block, or a block yet to be coded. With this configuration, thefirst derivation unit 612 can derive first merging candidates fromblocks appropriate for obtaining merging candidates.

When a plurality of first merging candidates has been derived, thespecification unit 613 specifies an identical candidate, that is, afirst merging candidate which is a combination of a predictiondirection, a motion vector, and a reference picture index identical to acombination of a prediction direction, a motion vector, and a referencepicture index of any other of the derived first merging candidates.Then, the specification unit 613 removes the specified identicalcandidate from the merging candidate list.

The second determination unit 614 determines whether or not the totalnumber of the first merging candidates is smaller than a determinedmaximum number. Here, the second determination unit 614 determineswhether or not the total number of the first merging candidates exceptthe specified identical first merging candidate is smaller than thedetermined maximum number.

When it is determined that the total number of the first mergingcandidates is smaller than the determined maximum number, the secondderivation unit 615 derives a second merging candidate forbi-directional prediction by making a combination out of the firstmerging candidates. Specifically, the second derivation unit 615 derivessecond merging candidates within a range in which the sum of the totalnumber of first merging candidates and the total number of the secondmerging candidates does not exceed the maximum number. Here, the secondderivation unit 615 derives second merging candidates within a range inwhich the sum of the total number of first merging candidates except theidentical candidate and the total number of the second mergingcandidates does not exceed the maximum number. Specifically, forexample, the second derivation unit 212 derives a second mergingcandidate in the same manner as the deriving of a combined merging blockcandidate in Embodiment 1 (see FIG. 15, etc.).

More specifically, the second derivation unit 615 derives a secondmerging candidate for bi-directional prediction by combining, forexample, a motion vector and a reference picture index for a firstprediction direction (prediction direction 0) which are included in oneof the first merging candidates and a motion vector and a referencepicture index for a second prediction direction (prediction direction 1)which are included in a different one of the first merging candidates.

Then, for example, the second derivation unit 615 registers secondmerging candidates derived in this manner in the merging candidate listeach in association with a different merging candidate index. At thistime, the second derivation unit 615 may register the second mergingcandidates in the merging candidate list so that the merging candidateindexes assigned to the first merging candidates are smaller than themerging candidate indexes assigned to the second merging candidates.With this, the image coding apparatus 600 can reduce the code amountwhen the first merging candidates are more likely to be selected as amerging candidate to be used for coding than a second merging candidateso that coding efficiency can be increased.

It should be noted that the second derivation unit 615 need not derive asecond merging candidate so that the sum of the total number of thefirst merging candidates and the total number of the second mergingcandidate equals a determined maximum number. When the sum of the totalnumber of the first merging candidates and the total number of thesecond merging candidate is smaller than the determined maximum number,for example, there may be a merging candidate index with which nomerging candidate is associated.

The prediction control unit 620 selects a merging candidate to be usedfor coding a current block from the first merging candidates and secondmerging candidates. In other words, the prediction control unit 620selects a merging candidate to be used for coding a current block fromthe merging candidate list.

The coding unit 630 codes the index for identifying the selected mergingcandidate (merging candidate index) using the determined maximum number.Specifically, the coding unit 630 performs variable-length coding on abit sequence assigned to the index value of the selected mergingcandidate as shown in FIG. 5. Furthermore, the coding unit 630 attachesthe coded index to a bitstream.

Here, the coding unit 630 may further attach information indicating themaximum number determined by the first determination unit 611 to thebitstream. Specifically, for example, the coding unit 630 may write theinformation indicating the maximum number in a slice header. This makesit possible to change maximum numbers by the appropriate unit so thatcoding efficiency can be increased.

The coding unit 630 need not attach a maximum number to a bitstream. Forexample, when the maximum number is specified in a standard, or when themaximum number is the same as a default value, the coding unit 630 neednot attach information indicating the maximum number to a bitstream.

Next, operations of the image coding apparatus 600 in theabove-described configuration will be described below.

FIG. 30 is a flowchart showing processing operations of the image codingapparatus 600 according to Embodiment 6.

First, the first determination unit 611 determines a maximum number ofmerging candidates (S601). The first derivation unit 612 derives firstmerging candidates (S602). The specification unit 613 specifies a firstmerging candidate which is an identical candidate, that is, acombination of a prediction direction, a motion vector, and a referencepicture index identical to a combination of a prediction direction, amotion vector, and a reference picture index of any other of the firstmerging candidates (S603).

The second determination unit 614 determines whether or not the totalnumber of the first merging candidates except the identical candidate issmaller than the determined maximum number (S604). Here, when it isdetermined that the total number of the first merging candidates exceptthe identical candidate is smaller than the determined maximum number(S604, Yes), the second derivation unit 615 derives a second mergingcandidates for bi-directional prediction by making a combination out ofthe first merging candidates (S605). On the other hand, when it isdetermined that the total number of the first merging candidates exceptthe identical candidate is not smaller than the determined maximumnumber (S604, No), the second derivation unit 615 derives no secondmerging candidate. These Step S604 and Step S605 correspond to Step S514in Embodiment 5.

The prediction control unit 620 selects a merging candidate to be usedfor coding of a current block from the first merging candidates andsecond merging candidates (S606). For example, the prediction controlunit 620 selects a merging candidate for which the cost represented byEquation 1 is a minimum from the merging candidate list as in Embodiment1.

The coding unit 630 codes an index for identifying the selected mergingcandidate, using the determined maximum number (S607). Furthermore, thecoding unit 630 attaches the coded index to a bitstream.

In this manner, the image coding apparatus 600 according to Embodiment 6is capable of deriving a second merging candidate for bi-directionalprediction by making a combination out of first merging candidatesderived based on blocks spatially or temporally neighboring a currentblock to be coded. In particular, the image coding apparatus 600 iscapable of deriving a second merging candidate for bi-directionalprediction even when the first merging candidates include nobi-directionally predicted merging candidate. As a result, the imagecoding apparatus 600 increases the variety of combinations of aprediction direction, a motion vector, and a reference picture indexfrom which a merging candidate is selected so that coding efficiency canbe increased.

Furthermore, the image coding apparatus 600 according to Embodiment 6can code an index for identifying a merging candidate using a determinedmaximum number. In other words, an index can be coded independently ofthe total number of actually derived merging candidates. Therefore, evenwhen information necessary for derivation of a merging candidate (forexample, information on a co-located block) is lost, an index can bestill decoded and error resistance is thereby enhanced. Furthermore, anindex can be decoded independently of the total number of actuallyderived merging candidates. In other words, an index can be decodedwithout waiting for derivation of merging candidates. In other words, abitstream can be generated for which deriving of merging candidates anddecoding of indexes can be performed in parallel.

Furthermore, with the image coding apparatus 600 according to Embodiment6, a second merging candidate can be derived when it is determined thatthe total number of the first merging candidates is smaller than themaximum number. Accordingly, the total number of merging candidates canbe increased within a range not exceeding the maximum number so thatcoding efficiency can be increased.

Furthermore, with the image coding apparatus 600 according to Embodiment6, a second merging candidate can be derived based on the total numberof first merging candidates except identical first merging candidates.As a result, the total number of the second merging candidates can beincreased so that the variety of combinations of a prediction direction,a motion vector, and a reference picture index for a selectable mergingcandidate can be increased. It is therefore possible to further increasecoding efficiency.

In Embodiment 6, the specification unit 613 included in the image codingapparatus 600 is not always necessary to the image coding apparatus 600.In other words, Step S603 in the flowchart shown in FIG. 30 is notalways necessary. Even in such a case, the image coding apparatus 600can code an index for identifying a merging candidate using a determinedmaximum number so that error resistance can be enhanced.

Furthermore, in Embodiment 6, although the specification unit 613specifies an identical candidate after the first derivation unit 612derives first merging candidates as shown in FIG. 30, the process neednot be performed in this order. For example, the first derivation unit612 may identify an identical candidate in the process for derivingfirst merging candidates, and derives the first merging candidates suchthat the specified identical candidate is excluded from the firstmerging candidates. In other words, the first derivation unit 612 mayderive, as a first merging candidate, a merging candidate which is acombination of a prediction direction, a motion vector, and a referencepicture index different from a combination of a prediction direction, amotion vector, and a reference picture index of any first mergingcandidate previously derived. More specifically, for example, in thecase where a merging candidate based on a left neighboring block hasalready been selected as a first merging candidate, the first derivationunit 612 may derive a merging candidate which is based on an upperneighboring block as a first merging candidate when the mergingcandidate based on the upper neighboring block is different from themerging candidate which is based on the left neighboring block.

In other words, the first derivation unit 612 may derive first mergingcandidates such that each of the first merging candidates is a differentcombination of a prediction direction, a motion vector, and a referencepicture index. In this manner, the first derivation unit 612 can remove,from the first merging candidates, a merging candidate which is acombination of a prediction direction, a motion vector, and a referencepicture index identical to a combination of a prediction direction, amotion vector, and a reference picture index of any first mergingcandidate previously derived. As a result, the image coding apparatus600 can increase the total number of the second merging candidates, andthereby increase the variety of combinations of a prediction direction,a motion vector, and a reference picture index from which a mergingcandidate is selected. The first derivation unit 612 thus can furtherincrease coding efficiency.

Embodiment 7

Embodiment 7 is different in the method of deriving the size of amerging block candidate list from Embodiment 3. The method of derivingthe size of a merging block candidate list according to Embodiment 7will be described below in detail.

FIG. 31 is a block diagram showing a configuration of an image decodingapparatus 700 according to Embodiment 7. For FIG. 31, the constituentelements in common with FIG. 19 are denoted with the same referencesigns, and description thereof is omitted.

The image decoding apparatus 700 is an apparatus corresponding to theimage coding apparatus 500 according to Embodiment 5. Specifically, forexample, the image decoding apparatus 700 decodes, on a block-by-blockbasis, coded images included in a bitstream generated by the imagecoding apparatus 500 according to Embodiment 5.

As shown in FIG. 31, the image decoding apparatus 700 includes avariable-length-decoding unit 701, an inverse-quantization unit 302, aninverse-orthogonal-transformation unit 303, an adder 304, block memory305, frame memory 306, an intra prediction unit 307, an inter predictionunit 308, an inter prediction control unit 309, a switch 310, a mergingblock candidate calculation unit 711, and colPic memory 312.

The variable-length-decoding unit 701 generates picture-typeinformation, a merging flag, and a quantized coefficient by performingvariable-length decoding on an input bitstream. Furthermore, thevariable-length-decoding unit 701 obtains a merging block candidateindex by performing variable-length decoding using the total number ofusable-for-merging candidates described below.

The merging block candidate calculation unit 711 derives merging blockcandidates for the merging mode from motion vectors and others ofneighboring blocks of the current block and a motion vector and othersof a co-located block (colPic information) stored in the colPic memory312, using a method described later. Furthermore, the merging blockcandidate calculation unit 711 assigns merging block candidate indexeseach having a different value to the derived merging block candidates.Then, the merging block candidate calculation unit 711 transmits themerging block candidates and merging block candidate indexes to theinter prediction control unit 309.

FIG. 32 is a flowchart showing processing operations of the imagedecoding apparatus according to Embodiment 7.

In Step S701, the variable-length-decoding unit 701 decodes a mergingflag.

In Step S702, when the merging flag is “1” (S702, Yes), in Step S703,the merging block candidate calculation unit 711 calculates the totalnumber of usable-for-merging candidates using a method described later.Then, the merging block candidate calculation unit 711 sets thecalculated number of usable-for-merging candidates as the size of amerging block candidate list.

Next, in Step S704, the variable-length-decoding unit 701 performsvariable-length decoding on a merging block candidate index from abitstream using the size of the merging block candidate list. In StepS705, the merging block candidate calculation unit 711 generates mergingblock candidates from neighboring blocks and a co-located block of acurrent block to be decoded using the method described in Embodiment 1or Embodiment 3 or a method described later.

In Step S706, the inter prediction control unit 309 causes the interprediction unit 308 to generate an inter prediction picture using themotion vector, reference picture index, and prediction direction of themerging block candidate indicated by the decoded merging block candidateindex.

When the merging flag is “0” in Step S702 (Step S702, No), in Step S707,the inter prediction unit 308 generates an inter prediction pictureusing information on motion vector estimation mode decoded by thevariable-length-decoding unit 701.

Optionally, when the size of a merging block candidate list calculatedin Step S703 is “1”, a merging block candidate index may be estimated tobe “0” without being decoded.

FIG. 33 is a flowchart showing details of the process in Step S703 shownin FIG. 32. Specifically, FIG. 33 illustrates a method of determiningwhether or not a merging block candidate [N] is a usable-for-mergingcandidate and calculating the total number of usable-for-mergingcandidates. FIG. 33 will be described below.

In Step S711, the merging block candidate calculation unit 711determines whether it is true or false that (1) a merging blockcandidate [N] has been decoded by intra prediction, (2) the mergingblock candidate [N] is a block outside the boundary of a slice includingthe current block or the boundary of a picture including the currentblock, or (3) the merging block candidate [N] is yet to be decoded.

When the result of the determination in Step S711 is true (S711, Yes),the merging block candidate calculation unit 711 sets the merging blockcandidate [N] as an unusable-for-merging candidate in Step S712. On theother hand, when the result of the determination in Step S711 is false(S711, No), the merging block candidate calculation unit 711 sets themerging block candidate [N] as a usable-for-merging candidate in StepS713.

In Step S714, the merging block candidate calculation unit 711determines whether it is true or false that the merging block candidate[N] is either a usable-for-merging candidate or a co-located mergingblock candidate. Here, when the result of the determination in Step S714is true (S714, Yes), the merging block candidate calculation unit 711updates the total number of merging block candidates by incrementing itby one in Step S715. On the other hand, when the result of thedetermination in Step S714 is false (S714, No), the merging blockcandidate calculation unit 711 does not update the total number ofusable-for-merging candidates.

Thus, when a merging block candidate is a co-located merging block, themerging block candidate calculation unit 711 increments the total numberof usable-for-merging candidates by one regardless of whether theco-located block is a usable-for-merging candidate or anunusable-for-merging candidate. This prevents discrepancy of the numbersof usable-for-merging candidates between the image coding apparatus andthe image decoding apparatus even when information on a co-locatedmerging block is lost due to an incident such as packet loss.

The total number of usable-for-merging candidates is set as the size ofa merging block candidate list in Step S703 shown in FIG. 32.Furthermore, the size of the merging block candidate list is used invariable-length decoding of merging block candidate indexes in Step S704shown in FIG. 32. This makes it possible for the image decodingapparatus 700 to decode merging block candidate indexes normally evenwhen information on reference picture including a co-located block islost.

FIG. 34 is a flowchart showing details of the process in Step S705 shownin FIG. 32. Specifically, FIG. 34 illustrates a method of calculating amerging block candidate. FIG. 34 will be described below.

In Step S721, the merging block candidate calculation unit 711 obtainsthe motion vector, reference picture index, and prediction direction ofa merging block candidate [N], and adds them to a merging blockcandidate list.

In Step S722, the merging block candidate calculation unit 711 searchesthe merging block candidate list for an unusable-for-merging candidateand an identical candidate, and removes the unusable-for-mergingcandidate and the identical candidate from the merging block candidatelist as shown in FIG. 25.

In Step S723, the merging block candidate calculation unit 711 adds anew candidate to the merging block candidate list using the methoddescribed in Embodiment 1 or Embodiment 3 or the method as illustratedin FIG. 28.

FIG. 35 shows exemplary syntax for attachment of merging block candidateindexes to a bitstream. In FIG. 35, merge_idx represents a merging blockcandidate index, and merge_flag represents a merging flag. NumMergeCandrepresents the size of a merging block candidate list. In Embodiment 7,NumMergeCand is set at the total number of usable-for-merging candidatescalculated in the process flow shown in FIG. 33.

Thus, the image decoding apparatus 700 according to Embodiment 7 iscapable of calculating the size of a merging block candidate list foruse in coding or decoding of a merging block candidate index, using amethod independent of information on reference pictures including aco-located block. The image decoding apparatus 700 therefore canappropriately decode a bitstream having enhanced error resistance.

More specifically, regardless of whether or not a co-located mergingblock is a usable-for-merging candidate, the image decoding apparatus700 according to Embodiment 7 increments the total number ofusable-for-merging candidates by one each time a merging block candidateis determined as a co-located merging block. Then, the image decodingapparatus 700 determines a bit sequence assigned to a merging blockcandidate index using the total number of usable-for-merging candidatescalculated in this manner. This makes it possible for the image decodingapparatus 700 to decode merging block candidate indexes normally evenwhen information on reference picture including a co-located block islost.

Furthermore, when the total number of the merging block candidates issmaller than the total number of the usable-for-merging candidates, itis possible for the image decoding apparatus 700 according to Embodiment7 to appropriately decode a bitstream coded with increased efficiency byadding a new candidate having a new motion vector, a new referencepicture index, and a new prediction direction.

Embodiment 8

In Embodiment 7, the image decoding apparatus determines a bit sequenceto be assigned to a merging block candidate index using the total numberof usable-for-merging candidates incremented by one each time a mergingblock candidate is determined as a co-located merging block, regardlessof whether or not a co-located merging block is a usable-for-mergingcandidate. Optionally, for example, the image decoding apparatus maydetermine a bit sequence to be assigned to a merging block candidateindex using the total number of usable-for-merging candidates calculatedby incrementing by one for each merging block candidate each mergingblock candidate regardless of whether or not the merging block candidateis a co-located merging block in Step S714 in FIG. 33. In other words,the image decoding apparatus may assign a bit sequence to a mergingblock candidate index using the size of a merging block candidate listfixed at a maximum number N of the total number of merging blockcandidates. In other words, the image decoding apparatus may decodemerging block candidate indexes using the size of a merging blockcandidate list fixed at a maximum value N of the total number of mergingblock candidates on the assumption that the merging block candidates areall usable-for-merging candidates.

For example, in the case shown in Embodiment 7, when the maximum value Nof the total number of merging block candidates is five (the neighboringblock A, neighboring block B, co-located merging block, neighboringblock C, and neighboring block D), the image decoding apparatus maydecode the merging block candidate indexes using the size of the mergingblock candidate list fixedly set at five. It is therefore possible forthe variable-length-decoding unit of the image decoding apparatus todecode a merging block candidate index from a bitstream withoutreferencing information on a neighboring block or on a co-located block.As a result, for example, Step S714 and Step S715 shown in FIG. 33 canbe skipped so that the computational complexity for thevariable-length-decoding unit can be reduced.

FIG. 36 shows exemplary syntax in the case where the size of a mergingblock candidate list is fixed at the maximum value of the total numberof merging block candidates. As can be seen in FIG. 36, NumMergeCand canbe omitted from the syntax when the size of a merging block candidatelist is fixed at a maximum value of the total number of merging blockcandidates.

Such a modification of the image decoding apparatus according toEmbodiment 7 will be specifically described below as an image decodingapparatus according to Embodiment 8.

FIG. 37 is a block diagram showing a configuration of an image decodingapparatus 800 according to Embodiment 8. An image decoding apparatus 800decodes a coded image included in a bitstream on a block-by-block basis.Specifically, for example, the image decoding apparatus 800 decodes, ona block-by-block basis, coded images included in a bitstream generatedby the image coding apparatus 600 according to Embodiment 6. The imagedecoding apparatus 800 includes a merging candidate derivation unit 810,a decoding unit 820, and a prediction control unit 830.

The merging candidate derivation unit 810 corresponds to the mergingblock candidate calculation unit 711 in Embodiment 7. The mergingcandidate derivation unit 810 derives merging candidates. The mergingcandidate derivation unit 810 generates a merging candidate list inwhich, for example, indexes each identifying a different derived mergingcandidate (merging candidate indexes) are associated with the respectivederived merging candidates.

As shown in FIG. 37, the merging candidate derivation unit 810 includesa first determination unit 811, a first derivation unit 812, aspecification unit 813, a second determination unit 814, and a secondderivation unit 815.

The first determination unit 811 determines a maximum number of mergingcandidates. In other words, the first determination unit 811 determinesa maximum value N of the total number of merging block candidates.

For example, the first determination unit 811 may determine a maximumnumber of the merging candidates using the same method used by the firstdetermination unit 611 in Embodiment 6. Optionally, for example, thefirst determination unit 811 may determine a maximum number based oninformation attached to a bitstream and indicating a maximum number. Theimage decoding apparatus 800 thus can decode an image coded usingmaximum numbers changed by the appropriate unit.

Here, although the first determination unit 811 is included in themerging candidate derivation unit 810, the first determination unit 811may be included in the decoding unit 820.

The first derivation unit 812 derives first merging candidates in thesame manner as the first derivation unit 612 in Embodiment 6.Specifically, the first derivation unit 812 derives first mergingcandidates based on, for example, prediction directions, motion vectors,and reference picture indexes which have been used in decoding blocksspatially or temporally neighboring a current block to be decoded. Then,for example, the first derivation unit 812 registers the first mergingcandidates derived in this manner in the merging candidate list each inassociation with a different merging candidate indexes.

It should be noted that the first derivation unit 812 may derive, as afirst merging candidate, a combination of a prediction direction, amotion vector, and a reference picture index which have been used indecoding blocks which spatially neighbor the current block exceptunusable-for-merging blocks. With this configuration, the firstderivation unit 812 can derive first merging candidates from blocksappropriate for obtaining merging candidates.

The specification unit 813 specifies an identical candidate, that is, afirst merging candidate which is a combination of a predictiondirection, a motion vector, and a reference picture index identical to acombination of a prediction direction, a motion vector, and a referencepicture index of any other of the derived first merging candidates.Then, the specification unit 813 removes the specified identicalcandidate from the merging candidate list.

The second determination unit 814 determines whether or not the totalnumber of the first merging candidates is smaller than a determinedmaximum number. Here, the second determination unit 814 determineswhether or not the total number of the first merging candidates exceptthe specified identical first merging candidate is smaller than thedetermined maximum number.

When it is determined that the total number of the first mergingcandidates is smaller than the determined maximum number, the secondderivation unit 815 derives a second merging candidate forbi-directional prediction by making a combination out of the firstmerging candidates. Specifically, the second derivation unit 815 derivessecond merging candidates in the same manner as the second derivationunit 615 in Embodiment 6. For example, the second derivation unit 815derives second merging candidates within a range in which the sum of thetotal number of first merging candidates and the total number of thesecond merging candidates does not exceed the maximum number. Here, thesecond derivation unit 815 derives second merging candidates within arange in which the sum of the total number of first merging candidatesexcept the identical candidate and the total number of the secondmerging candidates does not exceed the maximum number.

More specifically, the second derivation unit 815 derives a secondmerging candidate for bi-directional prediction by combining, forexample, a motion vector and a reference picture index for a firstprediction direction (prediction direction 0) which are included in oneof the first merging candidates and a motion vector and a referencepicture index for a second prediction direction (prediction direction 1)which are included in a different one of the first merging candidates.

Then, for example, the second derivation unit 815 registers secondmerging candidates derived in this manner in the merging candidate listeach in association with a different merging candidate index. At thistime, the second derivation unit 815 may register the second mergingcandidates in the merging candidate list so that the merging candidateindexes assigned to the first merging candidates are smaller than themerging candidate indexes assigned to the second merging candidates. Inthis manner, the image decoding apparatus 800 can appropriately decode abitstream coded with increased efficiency.

It should be noted that the second derivation unit 815 need not derive asecond merging candidate so that the sum of the total number of thefirst merging candidates and the total number of the second mergingcandidate equals a determined maximum number. When the sum of the totalnumber of the first merging candidates and the total number of thesecond merging candidate is smaller than the determined maximum number,for example, there may be a merging candidate index with which nomerging candidate is associated.

The decoding unit 820 decodes an index coded and attached to abitstream, which is an index for identifying a merging candidate, usingthe determined maximum number.

The prediction control unit 830 selects, based on the decoded index, amerging candidate to be used for decoding a current block from the firstmerging candidates and second merging candidates. In other words, theprediction control unit 830 selects a merging candidate to be used fordecoding a current block from the merging candidate list.

Next, operations of the image decoding apparatus 800 in theabove-described configuration will be explained below.

FIG. 38 is a flowchart showing processing operations of the imagedecoding apparatus 800 according to Embodiment 8.

First, the first determination unit 811 determines a maximum number ofmerging candidates (S801). The first derivation unit 812 derives a firstmerging candidate (S802). When a plurality of first merging candidateshas been derived, the specification unit 813 specifies a first mergingcandidate which is an identical candidate, that is, a combination of aprediction direction, a motion vector, and a reference picture indexidentical to a combination of a prediction direction, a motion vector,and a reference picture index of any other of the first mergingcandidates (S803).

The second determination unit 814 determines whether or not the totalnumber of the first merging candidates except the identical candidate issmaller than the determined maximum number (S804). Here, when it isdetermined that the total number of the first merging candidates exceptthe identical candidate is smaller than the determined maximum number(S804, Yes), the second derivation unit 815 derives second mergingcandidates (S805). On the other hand, when it is determined that thetotal number of the first merging candidates except the identicalcandidate is not smaller than the determined maximum number (S804, No),the second derivation unit 815 derives no second merging candidate.

The decoding unit 820 decodes an index coded and attached to abitstream, which is an index for identifying a merging candidate, usingthe determined maximum number (S806).

The prediction control unit 830 selects, based on the decoded index, amerging candidate to be used for decoding a current block from the firstmerging candidates and second merging candidates (S807). For example,the prediction control unit 830 selects a merging candidate for whichthe cost represented by Equation 1 is a minimum from the mergingcandidate list as in Embodiment 1.

Although the process is performed such that the decoding an index (S806)is performed after a merging candidate is derived, the process need notbe performed in this order. For example, a merging candidate may bederived (S802 to S805) after decoding an index (S806). Optionally,decoding an index (S806) and deriving of a merging candidate (S802 toS805) may be performed in parallel. This increases processing speed fordecoding.

In this manner, the image decoding apparatus 800 according to Embodiment8 is capable of deriving a second merging candidate for bi-directionalprediction by making a combination out of the first merging candidatesderived based on blocks spatially or temporally neighboring a currentblock to be decoded. In particular, the image decoding apparatus 800 iscapable of deriving a second merging candidate for bi-directionalprediction even when the first merging candidates include nobi-directionally predicted merging candidate. As a result, the imagedecoding apparatus 800 increases the variety of combinations of aprediction direction, a motion vector, and a reference picture indexfrom which a merging candidate is selected so that a bitstream codedwith increased efficiency can be appropriately decoded.

Furthermore, the image decoding apparatus 800 according to Embodiment 8can decode an index for identifying a merging candidate, using adetermined maximum number. In other words, an index can be decodedindependently of the total number of actually derived mergingcandidates. Therefore, even when information necessary for derivation ofa merging candidate (for example, information on a co-located block) islost, the image decoding apparatus 800 still can decode an index, anderror resistance is thereby enhanced. Furthermore, the image decodingapparatus 800 can decode an index without waiting for derivation ofmerging candidates so that deriving of merging candidates and decodingof indexes can be performed in parallel.

Furthermore, the image decoding apparatus 800 according to Embodiment 8is capable of deriving a second merging candidate when it is determinedthat the total number of the first merging candidates is smaller than amaximum number. Accordingly, the image decoding apparatus 800 canincrease the total number of merging candidates within a range notexceeding the maximum number, and appropriately decode a bitstream codedwith increased efficiency.

Furthermore, the image decoding apparatus 800 according to Embodiment 8is capable of deriving a second merging candidate based on the totalnumber of first merging candidates except identical first mergingcandidates. As a result, the image decoding apparatus 800 can increasethe total number of the second merging candidates, and thereby increasethe variety of combinations of a prediction direction, a motion vector,and a reference picture index from which a merging candidate isselected. The image decoding apparatus 800 thus can appropriately decodea bitstream coded with further increased coding efficiency.

As in Embodiment 6, the specification unit 813 included in the imagedecoding apparatus 800 is not always necessary to the image decodingapparatus 800 in Embodiment 8. In other words, Step S803 in theflowchart shown in FIG. 38 is not always necessary. Even in such a case,the image decoding apparatus 800 can decode an index for identifying amerging candidate using a determined maximum number so that errorresistance can be enhanced.

Furthermore, in Embodiment 8, although the specification unit 813specifies an identical candidate after the first derivation unit 812derives first merging candidates as shown in FIG. 38, the process neednot be performed in this order. For example, the first derivation unit812 may derive, as a first merging candidate, a merging candidate whichis a combination of a prediction direction, a motion vector, and areference picture index different from a combination of a predictiondirection, a motion vector, and a reference picture index of any firstmerging candidate previously derived. In other words, the firstderivation unit 812 may derive first merging candidates such that eachof the first merging candidates is a different combination of aprediction direction, a motion vector, and a reference picture index. Inthis manner, the first derivation unit 812 can remove, from the firstmerging candidates, a merging candidate which is a combination of aprediction direction, a motion vector, and a reference picture indexidentical to a combination of a prediction direction, a motion vector,and a reference picture index of any first merging candidate previouslyderived. As a result, the image decoding apparatus 800 can increase thetotal number of the second merging candidates, and thereby increase thevariety of combinations of a prediction direction, a motion vector, anda reference picture index from which a merging candidate is selected.With this, the image decoding apparatus 800 thus can appropriatelydecode a bitstream coded with further increased coding efficiency.

Although the image coding apparatus and image decoding apparatusaccording to one or more aspects of the present invention have beendescribed based on the embodiments, the present invention is not limitedto the exemplary embodiments. Those skilled in the art will readilyappreciate that many modifications of the exemplary embodiments orembodiments in which the constituent elements of the exemplaryembodiments are combined are possible without materially departing fromthe novel teachings and advantages described in the present invention.All such modifications and embodiments are also within scopes of one ormore aspects of the present invention.

In the exemplary embodiments, each of the constituent elements may beimplemented as a piece of dedicated hardware or implemented by executinga software program appropriate for the constituent element. Theconstituent elements may be implemented by a program execution unit suchas a CPU or a processor which reads and executes a software programrecorded on a recording medium such as a hard disk or a semiconductormemory. Here, examples of the software program which implements theimage coding apparatus or image decoding apparatus in the embodimentsinclude a program as follows.

Specifically, the program causes a computer to execute a method which isan image coding method for coding an image on a block-by-block basis togenerate a bitstream, and the method includes: determining a maximumnumber of merging candidates each of which is a combination of aprediction direction, a motion vector, and a reference picture index foruse in coding of a current block; deriving a plurality of first mergingcandidates based on prediction directions, motion vectors, and referencepicture indexes used in coding of blocks spatially or temporallyneighboring the current block; determining whether or not a total numberof the derived first merging candidates is smaller than the maximumnumber; deriving, by making a combination out of the derived firstmerging candidates, a second merging candidate for bi-directionalprediction when it is determined that the total number of the derivedfirst merging candidates is smaller than the maximum number; selecting amerging candidate to be used for the coding of the current block fromthe derived first merging candidates and the derived second mergingcandidate; and coding, using the determined maximum number, an index foridentifying the selected merging candidate, and attaching the codedindex to the bitstream.

Furthermore, the program causes a computer to execute an image decodingmethod for decoding, on a block-by-block basis, a coded image includedin a bitstream, and the method includes: determining a maximum number ofmerging candidates each of which is a combination of a predictiondirection, a motion vector, and a reference picture index for use indecoding of a current block; deriving a plurality of first mergingcandidates based on prediction directions, motion vectors, and referencepicture indexes used in decoding of blocks spatially or temporallyneighboring the current block; determining whether or not a total numberof the derived first merging candidates is smaller than the maximumnumber; deriving, by making a combination out of the derived firstmerging candidates, a second merging candidate for bi-directionalprediction when it is determined that the total number of the derivedfirst merging candidates is smaller than the maximum number; decoding anindex coded and attached to the bitstream, using the determined maximumnumber, the index being an index for identifying a merging candidate;and selecting, based on the decoded index, a merging candidate to beused for the decoding of a current block, the merging candidate beingselected from the derived first merging candidates and the derivedsecond merging candidate.

Embodiment 9

The processing described in each of embodiments can be simplyimplemented in an independent computer system, by recording, in arecording medium, a program for implementing the configurations of themoving picture coding method (image coding method) and the movingpicture decoding method (image decoding method) described in each ofembodiments. The recording media may be any recording media as long asthe program can be recorded, such as a magnetic disk, an optical disk, amagnetic optical disk, an IC card, and a semiconductor memory.

Hereinafter, the applications to the moving picture coding method (imagecoding method) and the moving picture decoding method (image decodingmethod) described in each of embodiments and systems using thereof willbe described. The system has a feature of having an image coding anddecoding apparatus that includes an image coding apparatus using theimage coding method and an image decoding apparatus using the imagedecoding method. Other configurations in the system can be changed asappropriate depending on the cases.

FIG. 39 illustrates an overall configuration of a content providingsystem ex100 for implementing content distribution services. The areafor providing communication services is divided into cells of desiredsize, and base stations ex106, ex107, ex108, ex109, and ex110 which arefixed wireless stations are placed in each of the cells.

The content providing system ex100 is connected to devices, such as acomputer ex111, a personal digital assistant (PDA) ex112, a cameraex113, a cellular phone ex114 and a game machine ex115, via the Internetex101, an Internet service provider ex102, a telephone network ex104, aswell as the base stations ex106 to ex110, respectively.

However, the configuration of the content providing system ex100 is notlimited to the configuration shown in FIG. 39, and a combination inwhich any of the elements are connected is acceptable. In addition, eachdevice may be directly connected to the telephone network ex104, ratherthan via the base stations ex106 to ex100 which are the fixed wirelessstations. Furthermore, the devices may be interconnected to each othervia a short distance wireless communication and others.

The camera ex113, such as a digital video camera, is capable ofcapturing video. A camera ex116, such as a digital camera, is capable ofcapturing both still images and video. Furthermore, the cellular phoneex114 may be the one that meets any of the standards such as GlobalSystem for Mobile Communications (GSM) (registered trademark), CodeDivision Multiple Access (CDMA), Wideband-Code Division Multiple Access(W-CDMA), Long Term Evolution (LTE), and High Speed Packet Access(HSPA). Alternatively, the cellular phone ex114 may be a PersonalHandyphone System (PHS).

In the content providing system ex100, a streaming server ex103 isconnected to the camera ex113 and others via the telephone network ex104and the base station ex109, which enables distribution of images of alive show and others. In such a distribution, a content (for example,video of a music live show) captured by the user using the camera ex113is coded as described above in each of embodiments (i.e., the camerafunctions as the image coding apparatus according to an aspect of thepresent invention), and the coded content is transmitted to thestreaming server ex103. On the other hand, the streaming server ex103carries out stream distribution of the transmitted content data to theclients upon their requests. The clients include the computer ex111, thePDA ex112, the camera ex113, the cellular phone ex114, and the gamemachine ex115 that are capable of decoding the above-mentioned codeddata. Each of the devices that have received the distributed datadecodes and reproduces the coded data (i.e., functions as the imagedecoding apparatus according to an aspect of the present invention).

The captured data may be coded by the camera ex113 or the streamingserver ex103 that transmits the data, or the coding processes may beshared between the camera ex113 and the streaming server ex103.Similarly, the distributed data may be decoded by the clients or thestreaming server ex103, or the decoding processes may be shared betweenthe clients and the streaming server ex103. Furthermore, the data of thestill images and video captured by not only the camera ex113 but alsothe camera ex116 may be transmitted to the streaming server ex103through the computer ex111. The coding processes may be performed by thecamera ex116, the computer ex111, or the streaming server ex103, orshared among them.

Furthermore, the coding and decoding processes may be performed by anLSI ex500 generally included in each of the computer ex111 and thedevices. The LSI ex500 may be configured of a single chip or a pluralityof chips. Software for coding and decoding video may be integrated intosome type of a recording medium (such as a CD-ROM, a flexible disk, anda hard disk) that is readable by the computer ex111 and others, and thecoding and decoding processes may be performed using the software.Furthermore, when the cellular phone ex114 is equipped with a camera,the video data obtained by the camera may be transmitted. The video datais data coded by the LSI ex500 included in the cellular phone ex114.

Furthermore, the streaming server ex103 may be composed of servers andcomputers, and may decentralize data and process the decentralized data,record, or distribute data.

As described above, the clients may receive and reproduce the coded datain the content providing system ex100. In other words, the clients canreceive and decode information transmitted by the user, and reproducethe decoded data in real time in the content providing system ex100, sothat the user who does not have any particular right and equipment canimplement personal broadcasting.

Aside from the example of the content providing system ex100, at leastone of the moving picture coding apparatus (image coding apparatus) andthe moving picture decoding apparatus (image decoding apparatus)described in each of embodiments may be implemented in a digitalbroadcasting system ex200 illustrated in FIG. 40. More specifically, abroadcast station ex201 communicates or transmits, via radio waves to abroadcast satellite ex202, multiplexed data obtained by multiplexingaudio data and others onto video data. The video data is data coded bythe moving picture coding method described in each of embodiments (i.e.,data coded by the image coding apparatus according to an aspect of thepresent invention). Upon receipt of the multiplexed data, the broadcastsatellite ex202 transmits radio waves for broadcasting. Then, a home-useantenna ex204 with a satellite broadcast reception function receives theradio waves. Next, a device such as a television (receiver) ex300 and aset top box (STB) ex217 decodes the received multiplexed data, andreproduces the decoded data (i.e., functions as the image decodingapparatus according to an aspect of the present invention).

Furthermore, a reader/recorder ex218 (i) reads and decodes themultiplexed data recorded on a recording medium ex215, such as a DVD anda BD, or (i) codes video signals in the recording medium ex215, and insome cases, writes data obtained by multiplexing an audio signal on thecoded data. The reader/recorder ex218 can include the moving picturedecoding apparatus or the moving picture coding apparatus as shown ineach of embodiments. In this case, the reproduced video signals aredisplayed on the monitor ex219, and can be reproduced by another deviceor system using the recording medium ex215 on which the multiplexed datais recorded. It is also possible to implement the moving picturedecoding apparatus in the set top box ex217 connected to the cable ex203for a cable television or to the antenna ex204 for satellite and/orterrestrial broadcasting, so as to display the video signals on themonitor ex219 of the television ex300. The moving picture decodingapparatus may be implemented not in the set top box but in thetelevision ex300.

FIG. 41 illustrates the television (receiver) ex300 that uses the movingpicture coding method and the moving picture decoding method describedin each of embodiments. The television ex300 includes: a tuner ex301that obtains or provides multiplexed data obtained by multiplexing audiodata onto video data, through the antenna ex204 or the cable ex203, etc.that receives a broadcast; a modulation/demodulation unit ex302 thatdemodulates the received multiplexed data or modulates data intomultiplexed data to be supplied outside; and amultiplexing/demultiplexing unit ex303 that demultiplexes the modulatedmultiplexed data into video data and audio data, or multiplexes videodata and audio data coded by a signal processing unit ex306 into data.

The television ex300 further includes: a signal processing unit ex306including an audio signal processing unit ex304 and a video signalprocessing unit ex305 that decode audio data and video data and codeaudio data and video data, respectively (which function as the imagecoding apparatus and the image decoding apparatus according to theaspects of the present invention); and an output unit ex309 including aspeaker ex307 that provides the decoded audio signal, and a display unitex308 that displays the decoded video signal, such as a display.Furthermore, the television ex300 includes an interface unit ex317including an operation input unit ex312 that receives an input of a useroperation. Furthermore, the television ex300 includes a control unitex310 that controls overall each constituent element of the televisionex300, and a power supply circuit unit ex311 that supplies power to eachof the elements. Other than the operation input unit ex312, theinterface unit ex317 may include: a bridge ex313 that is connected to anexternal device, such as the reader/recorder ex218; a slot unit ex314for enabling attachment of the recording medium ex216, such as an SDcard; a driver ex315 to be connected to an external recording medium,such as a hard disk; and a modem ex316 to be connected to a telephonenetwork. Here, the recording medium ex216 can electrically recordinformation using a non-volatile/volatile semiconductor memory elementfor storage. The constituent elements of the television ex300 areconnected to each other through a synchronous bus.

First, the configuration in which the television ex300 decodesmultiplexed data obtained from outside through the antenna ex204 andothers and reproduces the decoded data will be described. In thetelevision ex300, upon a user operation through a remote controllerex220 and others, the multiplexing/demultiplexing unit ex303demultiplexes the multiplexed data demodulated by themodulation/demodulation unit ex302, under control of the control unitex310 including a CPU. Furthermore, the audio signal processing unitex304 decodes the demultiplexed audio data, and the video signalprocessing unit ex305 decodes the demultiplexed video data, using thedecoding method described in each of embodiments, in the televisionex300. The output unit ex309 provides the decoded video signal and audiosignal outside, respectively. When the output unit ex309 provides thevideo signal and the audio signal, the signals may be temporarily storedin buffers ex318 and ex319, and others so that the signals arereproduced in synchronization with each other. Furthermore, thetelevision ex300 may read multiplexed data not through a broadcast andothers but from the recording media ex215 and ex216, such as a magneticdisk, an optical disk, and a SD card. Next, a configuration in which thetelevision ex300 codes an audio signal and a video signal, and transmitsthe data outside or writes the data on a recording medium will bedescribed. In the television ex300, upon a user operation through theremote controller ex220 and others, the audio signal processing unitex304 codes an audio signal, and the video signal processing unit ex305codes a video signal, under control of the control unit ex310 using thecoding method described in each of embodiments. Themultiplexing/demultiplexing unit ex303 multiplexes the coded videosignal and audio signal, and provides the resulting signal outside. Whenthe multiplexing/demultiplexing unit ex303 multiplexes the video signaland the audio signal, the signals may be temporarily stored in thebuffers ex320 and ex321, and others so that the signals are reproducedin synchronization with each other. Here, the buffers ex318, ex319,ex320, and ex321 may be plural as illustrated, or at least one buffermay be shared in the television ex300. Furthermore, data may be storedin a buffer so that the system overflow and underflow may be avoidedbetween the modulation/demodulation unit ex302 and themultiplexing/demultiplexing unit ex303, for example.

Furthermore, the television ex300 may include a configuration forreceiving an AV input from a microphone or a camera other than theconfiguration for obtaining audio and video data from a broadcast or arecording medium, and may code the obtained data. Although thetelevision ex300 can code, multiplex, and provide outside data in thedescription, it may be capable of only receiving, decoding, andproviding outside data but not the coding, multiplexing, and providingoutside data.

Furthermore, when the reader/recorder ex218 reads or writes multiplexeddata from or on a recording medium, one of the television ex300 and thereader/recorder ex218 may decode or code the multiplexed data, and thetelevision ex300 and the reader/recorder ex218 may share the decoding orcoding.

As an example, FIG. 42 illustrates a configuration of an informationreproducing/recording unit ex400 when data is read or written from or onan optical disk. The information reproducing/recording unit ex400includes constituent elements ex401, ex402, ex403, ex404, ex405, ex406,and ex407 to be described hereinafter. The optical head ex401 irradiatesa laser spot in a recording surface of the recording medium ex215 thatis an optical disk to write information, and detects reflected lightfrom the recording surface of the recording medium ex215 to read theinformation. The modulation recording unit ex402 electrically drives asemiconductor laser included in the optical head ex401, and modulatesthe laser light according to recorded data. The reproductiondemodulating unit ex403 amplifies a reproduction signal obtained byelectrically detecting the reflected light from the recording surfaceusing a photo detector included in the optical head ex401, anddemodulates the reproduction signal by separating a signal componentrecorded on the recording medium ex215 to reproduce the necessaryinformation. The buffer ex404 temporarily holds the information to berecorded on the recording medium ex215 and the information reproducedfrom the recording medium ex215. The disk motor ex405 rotates therecording medium ex215. The servo control unit ex406 moves the opticalhead ex401 to a predetermined information track while controlling therotation drive of the disk motor ex405 so as to follow the laser spot.The system control unit ex407 controls overall the informationreproducing/recording unit ex400. The reading and writing processes canbe implemented by the system control unit ex407 using variousinformation stored in the buffer ex404 and generating and adding newinformation as necessary, and by the modulation recording unit ex402,the reproduction demodulating unit ex403, and the servo control unitex406 that record and reproduce information through the optical headex401 while being operated in a coordinated manner. The system controlunit ex407 includes, for example, a microprocessor, and executesprocessing by causing a computer to execute a program for read andwrite.

Although the optical head ex401 irradiates a laser spot in thedescription, it may perform high-density recording using near fieldlight.

FIG. 43 illustrates the recording medium ex215 that is the optical disk.On the recording surface of the recording medium ex215, guide groovesare spirally formed, and an information track ex230 records, in advance,address information indicating an absolute position on the diskaccording to change in a shape of the guide grooves. The addressinformation includes information for determining positions of recordingblocks ex231 that are a unit for recording data. Reproducing theinformation track ex230 and reading the address information in anapparatus that records and reproduces data can lead to determination ofthe positions of the recording blocks. Furthermore, the recording mediumex215 includes a data recording area ex233, an inner circumference areaex232, and an outer circumference area ex234. The data recording areaex233 is an area for use in recording the user data. The innercircumference area ex232 and the outer circumference area ex234 that areinside and outside of the data recording area ex233, respectively arefor specific use except for recording the user data. The informationreproducing/recording unit 400 reads and writes coded audio, coded videodata, or multiplexed data obtained by multiplexing the coded audio andvideo data, from and on the data recording area ex233 of the recordingmedium ex215.

Although an optical disk having a layer, such as a DVD and a BD isdescribed as an example in the description, the optical disk is notlimited to such, and may be an optical disk having a multilayerstructure and capable of being recorded on a part other than thesurface. Furthermore, the optical disk may have a structure formultidimensional recording/reproduction, such as recording ofinformation using light of colors with different wavelengths in the sameportion of the optical disk and for recording information havingdifferent layers from various angles.

Furthermore, a car ex210 having an antenna ex205 can receive data fromthe satellite ex202 and others, and reproduce video on a display devicesuch as a car navigation system ex211 set in the car ex210, in thedigital broadcasting system ex200. Here, a configuration of the carnavigation system ex211 will be a configuration, for example, includinga GPS receiving unit from the configuration illustrated in FIG. 41. Thesame will be true for the configuration of the computer ex111, thecellular phone ex114, and others.

FIG. 44A illustrates the cellular phone ex114 that uses the movingpicture coding method and the moving picture decoding method describedin embodiments. The cellular phone ex114 includes: an antenna ex350 fortransmitting and receiving radio waves through the base station ex110; acamera unit ex365 capable of capturing moving and still images; and adisplay unit ex358 such as a liquid crystal display for displaying thedata such as decoded video captured by the camera unit ex365 or receivedby the antenna ex350. The cellular phone ex114 further includes: a mainbody unit including an operation key unit ex366; an audio output unitex357 such as a speaker for output of audio; an audio input unit ex356such as a microphone for input of audio; a memory unit ex367 for storingcaptured video or still pictures, recorded audio, coded or decoded dataof the received video, the still pictures, e-mails, or others; and aslot unit ex364 that is an interface unit for a recording medium thatstores data in the same manner as the memory unit ex367.

Next, an example of a configuration of the cellular phone ex114 will bedescribed with reference to FIG. 44B. In the cellular phone ex114, amain control unit ex360 designed to control overall each unit of themain body including the display unit ex358 as well as the operation keyunit ex366 is connected mutually, via a synchronous bus ex370, to apower supply circuit unit ex361, an operation input control unit ex362,a video signal processing unit ex355, a camera interface unit ex363, aliquid crystal display (LCD) control unit ex359, amodulation/demodulation unit ex352, a multiplexing/demultiplexing unitex353, an audio signal processing unit ex354, the slot unit ex364, andthe memory unit ex367.

When a call-end key or a power key is turned ON by a user's operation,the power supply circuit unit ex361 supplies the respective units withpower from a battery pack so as to activate the cell phone ex114.

In the cellular phone ex114, the audio signal processing unit ex354converts the audio signals collected by the audio input unit ex356 invoice conversation mode into digital audio signals under the control ofthe main control unit ex360 including a CPU, ROM, and RAM. Then, themodulation/demodulation unit ex352 performs spread spectrum processingon the digital audio signals, and the transmitting and receiving unitex351 performs digital-to-analog conversion and frequency conversion onthe data, so as to transmit the resulting data via the antenna ex350.Also, in the cellular phone ex114, the transmitting and receiving unitex351 amplifies the data received by the antenna ex350 in voiceconversation mode and performs frequency conversion and theanalog-to-digital conversion on the data. Then, themodulation/demodulation unit ex352 performs inverse spread spectrumprocessing on the data, and the audio signal processing unit ex354converts it into analog audio signals, so as to output them via theaudio output unit ex357.

Furthermore, when an e-mail in data communication mode is transmitted,text data of the e-mail inputted by operating the operation key unitex366 and others of the main body is sent out to the main control unitex360 via the operation input control unit ex362. The main control unitex360 causes the modulation/demodulation unit ex352 to perform spreadspectrum processing on the text data, and the transmitting and receivingunit ex351 performs the digital-to-analog conversion and the frequencyconversion on the resulting data to transmit the data to the basestation ex110 via the antenna ex350. When an e-mail is received,processing that is approximately inverse to the processing fortransmitting an e-mail is performed on the received data, and theresulting data is provided to the display unit ex358.

When video, still images, or video and audio in data communication modeis or are transmitted, the video signal processing unit ex355 compressesand codes video signals supplied from the camera unit ex365 using themoving picture coding method shown in each of embodiments (i.e.,functions as the image coding apparatus according to the aspect of thepresent invention), and transmits the coded video data to themultiplexing/demultiplexing unit ex353. In contrast, during when thecamera unit ex365 captures video, still images, and others, the audiosignal processing unit ex354 codes audio signals collected by the audioinput unit ex356, and transmits the coded audio data to themultiplexing/demultiplexing unit ex353.

The multiplexing/demultiplexing unit ex353 multiplexes the coded videodata supplied from the video signal processing unit ex355 and the codedaudio data supplied from the audio signal processing unit ex354, using apredetermined method. Then, the modulation/demodulation unit(modulation/demodulation circuit unit) ex352 performs spread spectrumprocessing on the multiplexed data, and the transmitting and receivingunit ex351 performs digital-to-analog conversion and frequencyconversion on the data so as to transmit the resulting data via theantenna ex350.

When receiving data of a video file which is linked to a Web page andothers in data communication mode or when receiving an e-mail with videoand/or audio attached, in order to decode the multiplexed data receivedvia the antenna ex350, the multiplexing/demultiplexing unit ex353demultiplexes the multiplexed data into a video data bit stream and anaudio data bit stream, and supplies the video signal processing unitex355 with the coded video data and the audio signal processing unitex354 with the coded audio data, through the synchronous bus ex370. Thevideo signal processing unit ex355 decodes the video signal using amoving picture decoding method corresponding to the moving picturecoding method shown in each of embodiments (i.e., functions as the imagedecoding apparatus according to the aspect of the present invention),and then the display unit ex358 displays, for instance, the video andstill images included in the video file linked to the Web page via theLCD control unit ex359. Furthermore, the audio signal processing unitex354 decodes the audio signal, and the audio output unit ex357 providesthe audio.

Furthermore, similarly to the television ex300, a terminal such as thecellular phone ex114 probably have 3 types of implementationconfigurations including not only (i) a transmitting and receivingterminal including both a coding apparatus and a decoding apparatus, butalso (ii) a transmitting terminal including only a coding apparatus and(iii) a receiving terminal including only a decoding apparatus. Althoughthe digital broadcasting system ex200 receives and transmits themultiplexed data obtained by multiplexing audio data onto video data inthe description, the multiplexed data may be data obtained bymultiplexing not audio data but character data related to video ontovideo data, and may be not multiplexed data but video data itself.

As such, the moving picture coding method and the moving picturedecoding method in each of embodiments can be used in any of the devicesand systems described. Thus, the advantages described in each ofembodiments can be obtained.

Furthermore, the present invention is not limited to embodiments, andvarious modifications and revisions are possible without departing fromthe scope of the present invention.

Embodiment 10

Video data can be generated by switching, as necessary, between (i) themoving picture coding method or the moving picture coding apparatusshown in each of embodiments and (ii) a moving picture coding method ora moving picture coding apparatus in conformity with a differentstandard, such as MPEG-2, MPEG-4 AVC, and VC-1.

Here, when a plurality of video data that conforms to the differentstandards is generated and is then decoded, the decoding methods need tobe selected to conform to the different standards. However, since towhich standard each of the plurality of the video data to be decodedconforms cannot be detected, there is a problem that an appropriatedecoding method cannot be selected.

In order to solve the problem, multiplexed data obtained by multiplexingaudio data and others onto video data has a structure includingidentification information indicating to which standard the video dataconforms. The specific structure of the multiplexed data including thevideo data generated in the moving picture coding method and by themoving picture coding apparatus shown in each of embodiments will behereinafter described. The multiplexed data is a digital stream in theMPEG-2 Transport Stream format.

FIG. 45 illustrates a structure of the multiplexed data. As illustratedin FIG. 45, the multiplexed data can be obtained by multiplexing atleast one of a video stream, an audio stream, a presentation graphicsstream (PG), and an interactive graphics stream. The video streamrepresents primary video and secondary video of a movie, the audiostream (IG) represents a primary audio part and a secondary audio partto be mixed with the primary audio part, and the presentation graphicsstream represents subtitles of the movie. Here, the primary video isnormal video to be displayed on a screen, and the secondary video isvideo to be displayed on a smaller window in the primary video.Furthermore, the interactive graphics stream represents an interactivescreen to be generated by arranging the GUI components on a screen. Thevideo stream is coded in the moving picture coding method or by themoving picture coding apparatus shown in each of embodiments, or in amoving picture coding method or by a moving picture coding apparatus inconformity with a conventional standard, such as MPEG-2, MPEG-4 AVC, andVC-1. The audio stream is coded in accordance with a standard, such asDolby-AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, and linear PCM.

Each stream included in the multiplexed data is identified by PID. Forexample, 0x1011 is allocated to the video stream to be used for video ofa movie, 0x1100 to 0x111F are allocated to the audio streams, 0x1200 to0x121F are allocated to the presentation graphics streams, 0x1400 to0x141F are allocated to the interactive graphics streams, 0x1B00 to0x1B1F are allocated to the video streams to be used for secondary videoof the movie, and 0x1A00 to 0x1A1F are allocated to the audio streams tobe used for the secondary audio to be mixed with the primary audio.

FIG. 46 schematically illustrates how data is multiplexed. First, avideo stream ex235 composed of video frames and an audio stream ex238composed of audio frames are transformed into a stream of PES packetsex236 and a stream of PES packets ex239, and further into TS packetsex237 and TS packets ex240, respectively. Similarly, data of apresentation graphics stream ex241 and data of an interactive graphicsstream ex244 are transformed into a stream of PES packets ex242 and astream of PES packets ex245, and further into TS packets ex243 and TSpackets ex246, respectively. These TS packets are multiplexed into astream to obtain multiplexed data ex247.

FIG. 47 illustrates how a video stream is stored in a stream of PESpackets in more detail. The first bar in FIG. 47 shows a video framestream in a video stream. The second bar shows the stream of PESpackets. As indicated by arrows denoted as yy1, yy2, yy3, and yy4 inFIG. 47, the video stream is divided into pictures as I pictures, Bpictures, and P pictures each of which is a video presentation unit, andthe pictures are stored in a payload of each of the PES packets. Each ofthe PES packets has a PES header, and the PES header stores aPresentation Time-Stamp (PTS) indicating a display time of the picture,and a Decoding Time-Stamp (DTS) indicating a decoding time of thepicture.

FIG. 48 illustrates a format of TS packets to be finally written on themultiplexed data. Each of the TS packets is a 188-byte fixed lengthpacket including a 4-byte TS header having information, such as a PIDfor identifying a stream and a 184-byte TS payload for storing data. ThePES packets are divided, and stored in the TS payloads, respectively.When a BD ROM is used, each of the TS packets is given a 4-byteTP_Extra_Header, thus resulting in 192-byte source packets. The sourcepackets are written on the multiplexed data. The TP_Extra_Header storesinformation such as an Arrival_Time_Stamp (ATS). The ATS shows atransfer start time at which each of the TS packets is to be transferredto a PID filter. The source packets are arranged in the multiplexed dataas shown at the bottom of FIG. 48. The numbers incrementing from thehead of the multiplexed data are called source packet numbers (SPNs).

Each of the TS packets included in the multiplexed data includes notonly streams of audio, video, subtitles and others, but also a ProgramAssociation Table (PAT), a Program Map Table (PMT), and a Program ClockReference (PCR). The PAT shows what a PID in a PMT used in themultiplexed data indicates, and a PID of the PAT itself is registered aszero. The PMT stores PIDs of the streams of video, audio, subtitles andothers included in the multiplexed data, and attribute information ofthe streams corresponding to the PIDs. The PMT also has variousdescriptors relating to the multiplexed data. The descriptors haveinformation such as copy control information showing whether copying ofthe multiplexed data is permitted or not. The PCR stores STC timeinformation corresponding to an ATS showing when the PCR packet istransferred to a decoder, in order to achieve synchronization between anArrival Time Clock (ATC) that is a time axis of ATSs, and an System TimeClock (STC) that is a time axis of PTSs and DTSs.

FIG. 49 illustrates the data structure of the PMT in detail. A PMTheader is disposed at the top of the PMT. The PMT header describes thelength of data included in the PMT and others. A plurality ofdescriptors relating to the multiplexed data is disposed after the PMTheader. Information such as the copy control information is described inthe descriptors. After the descriptors, a plurality of pieces of streaminformation relating to the streams included in the multiplexed data isdisposed. Each piece of stream information includes stream descriptorseach describing information, such as a stream type for identifying acompression codec of a stream, a stream PID, and stream attributeinformation (such as a frame rate or an aspect ratio). The streamdescriptors are equal in number to the number of streams in themultiplexed data.

When the multiplexed data is recorded on a recording medium and others,it is recorded together with multiplexed data information files.

Each of the multiplexed data information files is management informationof the multiplexed data as shown in FIG. 50. The multiplexed datainformation files are in one to one correspondence with the multiplexeddata, and each of the files includes multiplexed data information,stream attribute information, and an entry map.

As illustrated in FIG. 50, the multiplexed data information includes asystem rate, a reproduction start time, and a reproduction end time. Thesystem rate indicates the maximum transfer rate at which a system targetdecoder to be described later transfers the multiplexed data to a PIDfilter. The intervals of the ATSs included in the multiplexed data areset to not higher than a system rate. The reproduction start timeindicates a PTS in a video frame at the head of the multiplexed data. Aninterval of one frame is added to a PTS in a video frame at the end ofthe multiplexed data, and the PTS is set to the reproduction end time.

As shown in FIG. 51, a piece of attribute information is registered inthe stream attribute information, for each PID of each stream includedin the multiplexed data. Each piece of attribute information hasdifferent information depending on whether the corresponding stream is avideo stream, an audio stream, a presentation graphics stream, or aninteractive graphics stream. Each piece of video stream attributeinformation carries information including what kind of compression codecis used for compressing the video stream, and the resolution, aspectratio and frame rate of the pieces of picture data that is included inthe video stream. Each piece of audio stream attribute informationcarries information including what kind of compression codec is used forcompressing the audio stream, how many channels are included in theaudio stream, which language the audio stream supports, and how high thesampling frequency is. The video stream attribute information and theaudio stream attribute information are used for initialization of adecoder before the player plays back the information.

In the present embodiment, the multiplexed data to be used is of astream type included in the PMT. Furthermore, when the multiplexed datais recorded on a recording medium, the video stream attributeinformation included in the multiplexed data information is used. Morespecifically, the moving picture coding method or the moving picturecoding apparatus described in each of embodiments includes a step or aunit for allocating unique information indicating video data generatedby the moving picture coding method or the moving picture codingapparatus in each of embodiments, to the stream type included in the PMTor the video stream attribute information. With the configuration, thevideo data generated by the moving picture coding method or the movingpicture coding apparatus described in each of embodiments can bedistinguished from video data that conforms to another standard.

Furthermore, FIG. 52 illustrates steps of the moving picture decodingmethod according to the present embodiment. In Step exS100, the streamtype included in the PMT or the video stream attribute informationincluded in the multiplexed data information is obtained from themultiplexed data. Next, in Step exS101, it is determined whether or notthe stream type or the video stream attribute information indicates thatthe multiplexed data is generated by the moving picture coding method orthe moving picture coding apparatus in each of embodiments. When it isdetermined that the stream type or the video stream attributeinformation indicates that the multiplexed data is generated by themoving picture coding method or the moving picture coding apparatus ineach of embodiments, in Step exS102, decoding is performed by the movingpicture decoding method in each of embodiments. Furthermore, when thestream type or the video stream attribute information indicatesconformance to the conventional standards, such as MPEG-2, MPEG-4 AVC,and VC-1, in Step exS103, decoding is performed by a moving picturedecoding method in conformity with the conventional standards.

As such, allocating a new unique value to the stream type or the videostream attribute information enables determination whether or not themoving picture decoding method or the moving picture decoding apparatusthat is described in each of embodiments can perform decoding. Even whenmultiplexed data that conforms to a different standard is input, anappropriate decoding method or apparatus can be selected. Thus, itbecomes possible to decode information without any error. Furthermore,the moving picture coding method or apparatus, or the moving picturedecoding method or apparatus in the present embodiment can be used inthe devices and systems described above.

Embodiment 11

Each of the moving picture coding method, the moving picture codingapparatus, the moving picture decoding method, and the moving picturedecoding apparatus in each of embodiments is typically achieved in theform of an integrated circuit or a Large Scale Integrated (LSI) circuit.As an example of the LSI, FIG. 53 illustrates a configuration of the LSIex500 that is made into one chip. The LSI ex500 includes elements ex501,ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509 to bedescribed below, and the elements are connected to each other through abus ex510. The power supply circuit unit ex505 is activated by supplyingeach of the elements with power when the power supply circuit unit ex505is turned on.

For example, when coding is performed, the LSI ex500 receives an AVsignal from a microphone ex117, a camera ex113, and others through an AVIO ex509 under control of a control unit ex501 including a CPU ex502, amemory controller ex503, a stream controller ex504, and a drivingfrequency control unit ex512. The received AV signal is temporarilystored in an external memory ex511, such as an SDRAM. Under control ofthe control unit ex501, the stored data is segmented into data portionsaccording to the processing amount and speed to be transmitted to asignal processing unit ex507. Then, the signal processing unit ex507codes an audio signal and/or a video signal. Here, the coding of thevideo signal is the coding described in each of embodiments.Furthermore, the signal processing unit ex507 sometimes multiplexes thecoded audio data and the coded video data, and a stream 10 ex506provides the multiplexed data outside. The provided multiplexed data istransmitted to the base station ex107, or written on the recordingmedium ex215. When data sets are multiplexed, the data should betemporarily stored in the buffer ex508 so that the data sets aresynchronized with each other.

Although the memory ex511 is an element outside the LSI ex500, it may beincluded in the LSI ex500. The buffer ex508 is not limited to onebuffer, but may be composed of buffers. Furthermore, the LSI ex500 maybe made into one chip or a plurality of chips.

Furthermore, although the control unit ex501 includes the CPU ex502, thememory controller ex503, the stream controller ex504, the drivingfrequency control unit ex512, the configuration of the control unitex501 is not limited to such. For example, the signal processing unitex507 may further include a CPU. Inclusion of another CPU in the signalprocessing unit ex507 can improve the processing speed. Furthermore, asanother example, the CPU ex502 may serve as or be a part of the signalprocessing unit ex507, and, for example, may include an audio signalprocessing unit. In such a case, the control unit ex501 includes thesignal processing unit ex507 or the CPU ex502 including a part of thesignal processing unit ex507.

The name used here is LSI, but it may also be called IC, system LSI,super LSI, or ultra LSI depending on the degree of integration.

Moreover, ways to achieve integration are not limited to the LSI, and aspecial circuit or a general purpose processor and so forth can alsoachieve the integration. Field Programmable Gate Array (FPGA) that canbe programmed after manufacturing LSIs or a reconfigurable processorthat allows re-configuration of the connection or configuration of anLSI can be used for the same purpose.

In the future, with advancement in semiconductor technology, a brand-newtechnology may replace LSI. The functional blocks can be integratedusing such a technology. The possibility is that the present inventionis applied to biotechnology.

Embodiment 12

When video data generated in the moving picture coding method or by themoving picture coding apparatus described in each of embodiments isdecoded, compared to when video data that conforms to a conventionalstandard, such as MPEG-2, MPEG-4 AVC, and VC-1 is decoded, theprocessing amount probably increases. Thus, the LSI ex500 needs to beset to a driving frequency higher than that of the CPU ex502 to be usedwhen video data in conformity with the conventional standard is decoded.However, when the driving frequency is set higher, there is a problemthat the power consumption increases.

In order to solve the problem, the moving picture decoding apparatus,such as the television ex300 and the LSI ex500 is configured todetermine to which standard the video data conforms, and switch betweenthe driving frequencies according to the determined standard. FIG. 54illustrates a configuration ex800 in the present embodiment. A drivingfrequency switching unit ex803 sets a driving frequency to a higherdriving frequency when video data is generated by the moving picturecoding method or the moving picture coding apparatus described in eachof embodiments. Then, the driving frequency switching unit ex803instructs a decoding processing unit ex801 that executes the movingpicture decoding method described in each of embodiments to decode thevideo data. When the video data conforms to the conventional standard,the driving frequency switching unit ex803 sets a driving frequency to alower driving frequency than that of the video data generated by themoving picture coding method or the moving picture coding apparatusdescribed in each of embodiments. Then, the driving frequency switchingunit ex803 instructs the decoding processing unit ex802 that conforms tothe conventional standard to decode the video data.

More specifically, the driving frequency switching unit ex803 includesthe CPU ex502 and the driving frequency control unit ex512 in FIG. 53.Here, each of the decoding processing unit ex801 that executes themoving picture decoding method described in each of embodiments and thedecoding processing unit ex802 that conforms to the conventionalstandard corresponds to the signal processing unit ex507 in FIG. 53. TheCPU ex502 determines to which standard the video data conforms. Then,the driving frequency control unit ex512 determines a driving frequencybased on a signal from the CPU ex502. Furthermore, the signal processingunit ex507 decodes the video data based on the signal from the CPUex502. For example, the identification information described inEmbodiment 10 is probably used for identifying the video data. Theidentification information is not limited to the one described inEmbodiment 10 but may be any information as long as the informationindicates to which standard the video data conforms. For example, whenwhich standard video data conforms to can be determined based on anexternal signal for determining that the video data is used for atelevision or a disk, etc., the determination may be made based on suchan external signal. Furthermore, the CPU ex502 selects a drivingfrequency based on, for example, a look-up table in which the standardsof the video data are associated with the driving frequencies as shownin FIG. 56. The driving frequency can be selected by storing the look-uptable in the buffer ex508 and in an internal memory of an LSI, and withreference to the look-up table by the CPU ex502.

FIG. 55 illustrates steps for executing a method in the presentembodiment. First, in Step exS200, the signal processing unit ex507obtains identification information from the multiplexed data. Next, inStep exS201, the CPU ex502 determines whether or not the video data isgenerated by the coding method and the coding apparatus described ineach of embodiments, based on the identification information. When thevideo data is generated by the moving picture coding method and themoving picture coding apparatus described in each of embodiments, inStep exS202, the CPU ex502 transmits a signal for setting the drivingfrequency to a higher driving frequency to the driving frequency controlunit ex512. Then, the driving frequency control unit ex512 sets thedriving frequency to the higher driving frequency. On the other hand,when the identification information indicates that the video dataconforms to the conventional standard, such as MPEG-2, MPEG-4 AVC, andVC-1, in Step exS203, the CPU ex502 transmits a signal for setting thedriving frequency to a lower driving frequency to the driving frequencycontrol unit ex512. Then, the driving frequency control unit ex512 setsthe driving frequency to the lower driving frequency than that in thecase where the video data is generated by the moving picture codingmethod and the moving picture coding apparatus described in each ofembodiment.

Furthermore, along with the switching of the driving frequencies, thepower conservation effect can be improved by changing the voltage to beapplied to the LSI ex500 or an apparatus including the LSI ex500. Forexample, when the driving frequency is set lower, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set to a voltage lower than that in the case where the drivingfrequency is set higher.

Furthermore, when the processing amount for decoding is larger, thedriving frequency may be set higher, and when the processing amount fordecoding is smaller, the driving frequency may be set lower as themethod for setting the driving frequency. Thus, the setting method isnot limited to the ones described above. For example, when theprocessing amount for decoding video data in conformity with MPEG-4 AVCis larger than the processing amount for decoding video data generatedby the moving picture coding method and the moving picture codingapparatus described in each of embodiments, the driving frequency isprobably set in reverse order to the setting described above.

Furthermore, the method for setting the driving frequency is not limitedto the method for setting the driving frequency lower. For example, whenthe identification information indicates that the video data isgenerated by the moving picture coding method and the moving picturecoding apparatus described in each of embodiments, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set higher. When the identification information indicates thatthe video data conforms to the conventional standard, such as MPEG-2,MPEG-4 AVC, and VC-1, the voltage to be applied to the LSI ex500 or theapparatus including the LSI ex500 is probably set lower. As anotherexample, when the identification information indicates that the videodata is generated by the moving picture coding method and the movingpicture coding apparatus described in each of embodiments, the drivingof the CPU ex502 does not probably have to be suspended. When theidentification information indicates that the video data conforms to theconventional standard, such as MPEG-2, MPEG-4 AVC, and VC-1, the drivingof the CPU ex502 is probably suspended at a given time because the CPUex502 has extra processing capacity. Even when the identificationinformation indicates that the video data is generated by the movingpicture coding method and the moving picture coding apparatus describedin each of embodiments, in the case where the CPU ex502 has extraprocessing capacity, the driving of the CPU ex502 is probably suspendedat a given time. In such a case, the suspending time is probably setshorter than that in the case where when the identification informationindicates that the video data conforms to the conventional standard,such as MPEG-2, MPEG-4 AVC, and VC-1.

Accordingly, the power conservation effect can be improved by switchingbetween the driving frequencies in accordance with the standard to whichthe video data conforms. Furthermore, when the LSI ex500 or theapparatus including the LSI ex500 is driven using a battery, the batterylife can be extended with the power conservation effect.

Embodiment 13

There are cases where a plurality of video data that conforms todifferent standards, is provided to the devices and systems, such as atelevision and a cellular phone. In order to enable decoding theplurality of video data that conforms to the different standards, thesignal processing unit ex507 of the LSI ex500 needs to conform to thedifferent standards. However, the problems of increase in the scale ofthe circuit of the LSI ex500 and increase in the cost arise with theindividual use of the signal processing units ex507 that conform to therespective standards.

In order to solve the problem, what is conceived is a configuration inwhich the decoding processing unit for implementing the moving picturedecoding method described in each of embodiments and the decodingprocessing unit that conforms to the conventional standard, such asMPEG-2, MPEG-4 AVC, and VC-1 are partly shared. Ex900 in FIG. 57A showsan example of the configuration. For example, the moving picturedecoding method described in each of embodiments and the moving picturedecoding method that conforms to MPEG-4 AVC have, partly in common, thedetails of processing, such as entropy coding, inverse quantization,deblocking filtering, and motion compensated prediction. The details ofprocessing to be shared probably include use of a decoding processingunit ex902 that conforms to MPEG-4 AVC. In contrast, a dedicateddecoding processing unit ex901 is probably used for other processingunique to an aspect of the present invention. Since the aspect of thepresent invention is characterized by inverse quantization inparticular, for example, the dedicated decoding processing unit ex901 isused for inverse quantization. Otherwise, the decoding processing unitis probably shared for one of the entropy decoding, deblockingfiltering, and motion compensation, or all of the processing. Thedecoding processing unit for implementing the moving picture decodingmethod described in each of embodiments may be shared for the processingto be shared, and a dedicated decoding processing unit may be used forprocessing unique to that of MPEG-4 AVC.

Furthermore, ex1000 in FIG. 57B shows another example in that processingis partly shared. This example uses a configuration including adedicated decoding processing unit ex1001 that supports the processingunique to an aspect of the present invention, a dedicated decodingprocessing unit ex1002 that supports the processing unique to anotherconventional standard, and a decoding processing unit ex1003 thatsupports processing to be shared between the moving picture decodingmethod according to the aspect of the present invention and theconventional moving picture decoding method. Here, the dedicateddecoding processing units ex1001 and ex1002 are not necessarilyspecialized for the processing according to the aspect of the presentinvention and the processing of the conventional standard, respectively,and may be the ones capable of implementing general processing.Furthermore, the configuration of the present embodiment can beimplemented by the LSI ex500.

As such, reducing the scale of the circuit of an LSI and reducing thecost are possible by sharing the decoding processing unit for theprocessing to be shared between the moving picture decoding methodaccording to the aspect of the present invention and the moving picturedecoding method in conformity with the conventional standard.

INDUSTRIAL APPLICABILITY

The image coding method and image decoding method according to an aspectof the present disclosure is advantageously applicable to a movingpicture coding method and a moving picture decoding method.

The invention claimed is:
 1. An image decoding method for decoding acurrent block, comprising: deriving a first candidate having a firstmotion vector that has been used to decode a first block; deriving asecond candidate having a second motion vector that has been used todecode a second block, the second block being different from the firstblock; determining whether or not a total number of candidates havingthe first candidate and the second candidate is less than a maximumcandidate number; deriving a third candidate having a third motionvector and a fourth motion vector when the total number of candidateshaving the first candidate and the second candidate is less than themaximum candidate number, the third motion vector being identical to thefirst motion vector, the fourth motion vector being identical to thesecond motion vector; and decoding a coded index corresponding to acandidate having a motion vector, wherein the maximum candidate numberis used to decode the coded index, wherein a bitstream has the codedindex, wherein the motion vector is used to decode the current block,and wherein the candidate is one of a plurality of candidates having thefirst candidate, the second candidate, and the third candidate.
 2. Theimage decoding method according to claim 1, wherein, based on themaximum candidate number, a decoded index is determined from a bitsequence corresponding to the coded index.
 3. The image decoding methodaccording to claim 1, wherein the third candidate is different from oneof the first candidate and the second candidate.
 4. The image decodingmethod according to claim 1, wherein the deriving of the first candidateincludes deriving a plurality of first candidates, and the imagedecoding method further comprises: removing an identical candidate fromthe plurality of first candidates before the determining.
 5. The imagedecoding method according to claim 1, wherein the deriving of the firstcandidate includes deriving a plurality of first candidates, and theimage decoding method further comprises: removing anunusable-for-merging candidate from the plurality of first candidatesbefore the determining.
 6. An image decoding apparatus that decodes acurrent block, comprising: a first deriver configured to derive a firstcandidate having a first motion vector that has been used to decode afirst block; a second deriver configured to derive a second candidatehaving a second motion vector that has been used to decode a secondblock, the second block being different from the first block; adeterminer configured to determine whether or not a total number ofcandidates having the first candidate and the second candidate is lessthan a maximum candidate number; a third deriver configured to derive athird candidate having a third motion vector and a fourth motion vectorwhen the total number of candidates having the first candidate and thesecond candidate is less than the maximum candidate number, the thirdmotion vector being identical to the first motion vector, the fourthmotion vector being identical to the second motion vector; and a decoderconfigured to decode a coded index corresponding to a candidate having amotion vector, wherein the maximum candidate number is used to decodethe coded index, wherein a bitstream has the coded index, wherein themotion vector is used to decode the current block, wherein the candidateis one of a plurality of candidates having the first candidate, thesecond candidate, and the third candidate.